AARL

Research practices, evaluation and infrastructure in the digital environment

John W Houghton, with Colin Steele and Margaret Henty

Abstract This paper examines changing research practices in the digital environment and draws out implications for research evaluation and the development of research infrastructure. Reviews of the literature, quantitative indicators of research activities and our own field research in Australia suggest that a new mode of knowledge production is emerging, changing research practices and bringing new information access and dissemination needs. Adjustments will be required to accommodate these changes. New opportunities are emerging for more appropriate, cost-effective and sustainable information access and dissemination, but to realise these opportunities it will be necessary to take a more holistic approach to the scholarly communications system and focus greater attention on the structure of incentives operating within it.

This paper reports on a study that examined changing research practices in the digital environment and drew out implications for scholarly communication and the development of research support infrastructure. [1] The study took a two-pronged approach. First, wide-ranging statistical and literature reviews provided the framework and grounding for analysis. Second, in-depth interviews were structured with a small structured sample of senior Australian researchers in a range of fields and institutional settings, with findings tested and extended in a series of workshops. Both interviews and workshops targeted the leading edge of research and focused on the drivers of change in research practice.

The production of knowledge

There are various models that seek to shed light on the changing nature of research. Perhaps most prominent among recent approaches are:

• the systems of innovation approach, [2] which encompasses a wide range of work focusing on the systems within which knowledge is produced, communicated and applied,
• the new production of knowledge approach, [3] which compares and contrasts traditional disciplinary research (mode one) with an emerging transdisciplinary, problem oriented mode of knowledge production (mode two), and
• the triple helix approach, [4] which describes the emerging inter-relationship between universities, industry and the state.

There are many facets to these models, other models that deserve mention and numerous critiques. Nevertheless, they have common elements that point to key dimensions of change in research practice. The systems of innovation approach places knowledge production in a wider context, highlighting linkages between various actors and activities and the inter-relationship between knowledge production and its application. The new production of knowledge approach highlights differences between traditional and new modes of knowledge production. The triple helix approach highlights the convergence and cross-over of institutional spheres - where, for example, firms become research centres and found universities, while universities become more like firms. Importantly, they are all suggestive of a trajectory of change.

The direction of change identified by these approaches involves:

• increasing diversity in the location of research activities - with a greater range of organisations involved in research (eg universities, research institutes, hospitals, firms, industry associations, etc),
• increasing focus on interdisciplinary and transdisciplinary research - with teams of researchers coming together to work on common problems that cannot be tackled adequately within a single disciplinary framework (eg environment and health),
• increasing focus on problems, rather than techniques - with solutions being sought from a range of disciplinary 'toolboxes', but findings valued for their contribution to the solution, rather than to the toolbox,
• increasing blurring of organisational borders and greater emphasis on collaborative work and communication - with a more flexible team-based approach in which teams form around problems, and then break up and move on to form different teams around different problems,
• changes in the modes of communication - including some increase in commercial guarding of intellectual property, somewhat less emphasis on publication in refereed journals and more on diverse and informal communication through networks of researchers, and
• more diverse forms of accountability - with economic and social, as well as disciplinary outcomes taken into account, and quality judged on a broader range of criteria. [5]

A review of quantitative indicators of change in research activities in Australia reveals that research is being conducted in an increasingly diverse range of settings, with significant changes in the sources of funding and locations of expenditure. There has been a marked increase in R&D activity in the business sector in Australia and a relative decline in the government sector. Between 1968-1969 and 2000-2001, the business sector's expenditure on R&D in Australia increased from 33 per cent of total to 47 per cent, while that of government decreased from 42 per cent to 23 per cent.

There has been a shift away from pure/basic research towards more applied and developmental activities, and a decline in the share of expenditure on 'non-oriented' research combined with increases in the more interdisciplinary, problem oriented fields. In 1992-1993, experimental development accounted for 38 per cent of R&D expenditure in Australia, applied research for 33 per cent, strategic basic research for 16 per cent and pure basic research for 12 per cent. By 2000-2001, applied research had increased its share of total R&D expenditure to 36 per cent, while experimental development accounted for a steady 38 per cent. Both pure basic research and strategic basic shares declined. Analysis of R&D activities by socio-economic objective reveals a shift towards more problem-oriented research. Over the period 1986-1987 to 2000-2001, expenditure on R&D by socio-economic objective in Australia increased by six per cent per annum. Society and health (ten per cent per annum) and economic development (six per cent per annum) were the fastest growing areas, while expenditure on non-oriented research (broadly speaking, pure/basic research) increased by just 1.3 per cent per annum.

Figure 1
R&D expenditure by sector of performance, 1968-1969 and 2000-2001 (per cent)

Source: Australian Bureau of Statistics. Analysis conducted for J W Houghton et al. Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training, 2003

While it is still overwhelmingly the case that higher education R&D activity is focused on pure/basic research and business sector R&D activity is focused on experimental development, growth trends reflect a restructuring of activities and some diversification of the location of research. Both the business and government sectors appear to be increasing their involvement in basic research, while there is an increasing emphasis within higher education institutions on applied research and experimental development. While still small, these shifts could have a significant impact on researcher perceptions and on their research practices. [6]

Australian citation data reveal increasing output, increasing collaboration (co-authorship), and an internationalisation of research. Between 1981 and 1999, the number of Australian publications in the Science Citation Index increased 3.3 per cent per annum. Over that period, the number of publications with a single author decreased by 1.5 per cent per annum, papers with institutional collaboration increased three per cent per annum, papers with national collaboration increased 7.6 per cent per annum and papers with international collaboration increased 9.9 per cent per annum.[7]

Figure 2
Levels of collaboration (joint authorship) in Australian publications in the Science Citation Index, 1981 to 1999

Source: L Butler Monitoring Australia's Scientific Research Canberra Australian Academy of Science 2001. Analysis conducted for J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Australian patenting data show that patenting too has increased rapidly, with international patent applications increasing even more rapidly than domestic applications. During 1990-1991, IP Australia received a total of 27 529 patent applications. By 1999-2000, annual applications had increased to 69 645, or by almost eleven per cent per annum. Over the same period, international applications received by the patent office increased from 612 to 1718, or by more than twelve per cent per annum.[8]

Figure 3
Patent applications received by IP Australia, 1990-1991 to 1999-2000 (indexed)

Source: IP Australia. Analysis conducted for J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

All these trends indicate a wider range of research activities taking place in a wider range of settings, in both sectoral and geographic senses, and a marked increase in collaboration. There is, in short, evidence of the emergence of a new mode of knowledge production and of a triple helix style relationship between universities and research organisations, industry and the state.

Changing research practices

To further understand the direction and pace of these developments we conducted in-depth interviews and workshops, involving around 75 Australian-based researchers from a range of research fields and institutional settings (40 interviewees and 35 workshop participants). The approach was more ethnographic than statistical. Interviews targeted active, senior researchers able to talk authoritatively about their field of research, as well as their own work. This led to the selection of a sample that was more senior in terms of both position and age than would have been the case with a random sample. Just over 70 per cent of interviewees were based at university faculty, key or special centres, almost 20 per cent were based primarily at the CSIRO and the remaining ten per cent were operating primarily through Cooperative Research Centres (CRCs). By research cluster, 29 per cent of interviewees were in medical and biological sciences, 23 per cent in physical sciences and engineering, 23 per cent in arts and humanities, 15 per cent in social sciences and ten per cent in area studies and languages.

Both interviews and workshops were wide-ranging and exploratory, focusing on the drivers of change in research practice. Our findings built on a substantial corpus of knowledge (as outlined in the full report). Nevertheless, given the structure of the sample, the small sample size and the exploratory nature of the interviews, reported percentages should be taken as no more than a guide to what is happening. However, comments derived from the interviews and workshops shed considerable light on why it is happening, and extensive researcher comments were included in the full report in order to shed light on their thinking on particular issues and to provide a window onto their reasons for doing what they do.

Communication and collaboration

As elsewhere, [9] researchers in Australia are operating in an increasingly institutionally diverse and interdisciplinary environment, in which collaboration is encouraged, competition is intense, performance is closely monitored and both scholarly and commercial outcomes are expected. Collaboration on projects is widespread. Almost 75 per cent of the Australian researchers interviewed reported working primarily in project teams, and a further 13 per cent reported a mixture of team and individual work. Very few worked entirely alone.

There is increasing collaboration across a wider range of organisational settings. Almost 60 per cent reported an increase in the locational diversity of collaborators, more than 50 per cent reported an increase in inter-institutional collaboration, around 45 per cent reported an increase in the average size of research teams and 25 per cent reported an increase in their own collaboration. Collaboration also seems to be spreading into the humanities, with 50 per cent of respondents in social sciences, humanities and arts reporting an increase in their team participation. Fully 90 per cent saw themselves as part of a wider collegial network - 100 per cent in science and medical fields, and 80 per cent in social sciences, humanities and arts. More than 70 per cent said that these networks were 'very important' to their work, and a further 15-20 per cent said that they were 'important'.

A number of researchers see encouragement by funding agencies as a key driver of collaboration, with agencies seeking scale, quality and value for money. For some, collaboration is driven by a need to access intellectual property, data, specialist instruments, large scale facilities and sample populations for surveys and trials. Others see an increasing need to collaborate as projects demand a wider range of specialist skills, with collaboration a response to complexity, increasing interdisciplinarity and an increasingly problem oriented approach to research.

Asked whether the new networked information environment had changed the way they collaborate, around 60 per cent said that it had - with a higher proportion of social sciences, humanities and arts researchers reporting change than those in science and medical fields. The time involved in handling the volume of e-mail and related information, and some potential loss of the human touch in interactions were widely cited concerns. Others reflected more specifically on the impacts of electronic communication on their field of research, noting some danger in the western and/or English language dominance of the internet to date, both in terms of participation and perspective. Nevertheless, most took a positive view, with connectedness and the increased pace and immediacy of communication adding to the sense of excitement and involvement and facilitating participation in global research networks.

Information search and access

Peer reviewed journals, books and conference papers were the most important physical/print sources for the researchers we interviewed. Some 60 per cent regarded peer reviewed journals as 'essential'. Books were regarded as an 'essential' source by almost 50 per cent, with a further 45 per cent using them. Conference papers were regarded as 'essential' by around 30 per cent, with all respondents reporting using them. There were notable differences between research fields. In social science, humanities and the arts, books ranked higher than refereed journals (print), with 70 per cent regarding books as 'essential' and the remaining 30 per cent using them. Less than 30 per cent of science and medical researchers regarded books as 'essential' and 10 per cent seemed not to use them at all.

Figure 4
Rating of print and physical sources

Note: N=40
Source: J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Perhaps indicating a fundamental shift towards electronic sources, a number of researchers indicated that they used print sources to catch up with things outside their immediate field of interest and for teaching. For an increasing number, it seems, print sources are more of an historical record than the current dialogue.

Figure 5
Rating of electronic and online sources

Note: N=40
Source: J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Online, generic search engines (eg Google) were regarded as 'essential' by two-thirds of the researchers interviewed, with web browser bookmarks or favourites regarded as 'essential' by around 40 per cent. Peer reviewed journal papers in electronic form were regarded as 'essential' by around 60 per cent, and conference papers were regarded as 'essential' by almost 50 per cent. Online alerting services and e-mail based newsletter subscriptions were regarded as 'essential' by around 30 per cent, and discussion groups (eg listserv) were considered 'essential' by around 20 per cent. All appeared to be widely used.

Researchers in science and medical fields appear to put more weight on peer review than do their counterparts in the social sciences, humanities and arts - two-thirds suggested that peer review was 'very important' and the remaining third suggested that it was 'important', compared with 47 per cent and 42 per cent of researchers in social sciences, humanities and arts, respectively. Among humanities researchers, credibility (defined more broadly than peer review) was seen as a more important factor. Researchers in science and medical fields also appear to rate the availability of an electronic version, desktop access and the availability of full text somewhat more highly than do researchers in social sciences, humanities and arts.

Figure 6
Factors influencing choice of information sources

Note: N=40
Source: J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Almost a third of all respondents saw research databases as 'essential' and a further 35 per cent reported using them. Their use was higher in the sciences, with 75 per cent of science and medical researchers using databases, of which more than 40 per cent suggested that they were 'essential'. Nevertheless, almost 60 per cent of social science, humanities and arts researchers reported using databases, of which 15 per cent regarded them as 'essential'.

Just over 40 per cent of the researchers interviewed reported that they had perceived a change in the relative importance of primary source datasets and secondary sources - 37 per cent in social sciences, humanities and arts, and 47 per cent in science and medical fields. Asked about the nature of that change, as many as 80 per cent reported noticing an increase in the use of primary data - 100 per cent of those in science and medical fields and around 60 per cent of those in social sciences, humanities and arts. A number of researchers suggested that access to primary data, related software and computer facilities was changing their field of research in quite fundamental ways.

Dissemination and publication

A high level of collaboration is reflected in joint authorship, with more than 75 per cent of respondents indicating that they were typically joint authors - 100 per cent in science and medical fields, and around 50 per cent in social sciences, humanities and arts. Some 40 per cent of respondents suggested that their patterns of authorship had changed over the last five years, with all citing either an increase in the number of authors or in the proportion of their published output that is joint authored.

Researchers revealed a strong preference for publication in peer reviewed journals, with more than 90 per cent suggesting that refereed journals were their preferred medium for publication. Book chapters, books (authored or edited) and conference papers were each nominated as preferred media by around 20 per cent. Among researchers in science and medical fields there was a stronger preference for publishing in peer reviewed journals. All expressed such a preference, while just 15 per cent nominated conference papers. Perhaps surprisingly, more than 80 per cent of researchers in social sciences, humanities and arts expressed a preference for publication in refereed journals, with just 30-35 per cent citing a preference for publication in books.

Format preferences varied from medium to medium. Around 80 per cent said they would prefer their journal papers to be available in print and 100 per cent wanted their books and reports to be in print. Conversely, around 40 per cent expressed a preference for their conference papers to be available electronically. When discussing medium preferences it seemed that researchers publish in the medium and format recognised by their peers, funding agencies and employing institutions. Other factors in determining preferences related to establishing a 'portfolio' of publication to demonstrate a range of talents, focusing on the prestige of the journal title and impact factors and aiming for a specific audience (readership).

Figure 7
Preferred medium for publication: by field of research (percentage of expressed preferences)

Source: J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Exploring the reasons behind publication revealed that communication of work and measuring research performance were both considered 'very important' - by around 60 per cent of respondents - and accreditation and recognition (within the research community) was considered 'very important' by around half. There were some differences between fields of research, with communication of work rated 'very important' by less than 60 per cent of researchers in science and medical fields, and ranking third behind indicator of performance (rated 'very important' by 60 per cent) and accreditation and recognition (rated 'very important' by more than 70 per cent). By contrast, communication of work was rated 'very important' by almost 70 per cent of researchers in social sciences, humanities and arts, ahead of indicator of performance (rated 'very important' by more than 60 per cent) and accreditation and recognition (rated 'very important' by just 30 per cent).

Comments on the motivations for publishing focused on communication with specific audiences and performance evaluation. A number were critical of Australia's research evaluation system, because it is based primarily on quantity rather than quality and because of what does and does not count as an output. Knowledge production has become more diverse and is often more closely linked to its context of application, but performance indicators do not yet entirely reflect such changes.

Figure 8
Main reasons for publishing

Note: N=40
Source: J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Formal peer review and the perceived prestige of the outlet were the most important factors when choosing between publishing outlets, with breadth of exposure and speed of publication also important. Interestingly, the availability of an electronic version and the ability to retain copyright did not seem to be important considerations for most researchers. Peer review and the prestige of the outlet appear to be somewhat more important for researchers in science and medical fields. Other considerations appeared to rate much lower, with speed of publication and breadth of circulation each seen as 'very important' by around 20 per cent. By contrast, only just over 50 per cent of researchers in social sciences, humanities and arts indicated that the perceived prestige of the outlet and formal peer review were 'very important'. Breadth of exposure and speed of publication were correspondingly higher in their consideration set.

Most of the researchers we spoke to felt that electronic publishing alternatives were supplementing print, rather than replacing it. Nevertheless, some 40 per cent said that the networked information environment had changed the way that they and others in their field publish and disseminate findings. Others said that it had not really changed what they were doing: things were much easier and quicker, and they were now publishing online, but it was the electronic version of the same thing (eg similar papers in the same journals).

Figure 9
Factors in choosing between publishing outlets

Note: N=40
Source: J W Houghton et al Changing Research Practices in the Digital Information and Communication Environment Canberra Department of Education Science and Training 2003

Conclusions

A new mode of knowledge production is emerging. There is increasing diversity in the location of research activities; increasing focus on interdisciplinary, multidisciplinary and transdisciplinary research; increasing focus on problems, rather than techniques; greater emphasis on collaborative work and communication; and greater emphasis on more diverse and informal modes of communication. The existing research infrastructure evolved over many years, during which traditional disciplinary research was the dominant mode. Consequently, existing research infrastructure and research evaluation practices tend to be better suited to the traditional than to new modes of research.

The emergence of new modes of research is bringing new information access and dissemination needs. There is increasing demand for access to a wider range of more diverse sources; for access mechanisms that cut across disciplinary silos; and for access to, and management of, non-traditional, non-text digital objects. Research databases, related software and other analytical objects are now core tools, as the very nature of discourse shifts from hypothesis testing towards collecting, processing and analysing primary data. As the US National Research Council noted:

The rapidly expanding availability of primary sources of data in digital form may be shifting the balance of research away from working with secondary sources such as scholarly publications. Researchers today struggle to extract meaning from these masses of data, because our techniques of searching, analysing, interpreting, and certifying information remain primitive. New automated systems, and perhaps new intermediary institutions for searching and authenticating information, will develop to provide these services, much as libraries and scholarly publications served these roles in the past. [10]

New scholarly publishing methods and publisher 'business models' are emerging. New digital object access management systems will be required, and there will be increasing demand for collaborative research support applications.

Many factors have influenced the development of the present research infrastructure and scholarly communication system. Its elements have been instituted, and are often considered separately. As a result, developments have been somewhat piecemeal and there are sometimes conflicting forces at work. It will be essential to adopt a more holistic approach to 're-engineering' the system, which treats the creation, production and distribution of scientific and scholarly information, the management of information rights and access, intellectual property and the incentive system, systems of evaluation and the underlying infrastructure as parts of a single system. Research practices are shaped by evaluation, changing funding patterns and priorities. Existing evaluation and reward structures tend to lead to conflicting incentives, and establishing a coherent structure of incentives that operates system-wide will be an essential step towards providing more appropriate and cost-effective access and dissemination.

Scientific and scholarly publishing is now evolving along two distinct paths - one in which large multinational commercial publishers are increasing their dominance in such areas as branded journal titles and access to scientific publication, and the other in which there is a variety of open access initiatives emerging. Open access subject archives and institutional repositories, operating in parallel with existing commercial publishing mechanisms (eg subscription and open access journals), now provide an opportunity to develop a more flexible and sustainable information infrastructure for both traditional and emerging modes of research.

We concluded that to realise the opportunities it will be necessary to implement an agenda that focuses on:

• creating a coherent structure of incentives through reforms to research evaluation and intellectual property management based on an holistic approach to the system for the creation, production and distribution of research information,
• providing the infrastructure and tools to support collaborative research activities in both traditional and new modes of research,
• enabling access to necessary information access mechanisms and resources, and equipping users with appropriate skills to enable their use, and
• encouraging the development a system of scholarly communication and research dissemination built on the principle of open access.

Notes

1. J W Houghton, C Steele and M Henty Changing Research Practices in the Digital Information and Communication Environment Canberra: Department of Education, Science and Training 2003. The project was supported by funding from the Australian Department of Education Science and Training Canberra. The full report is available at http://www.dest.gov.au/highered/ respubs/changing_res_prac/exec_summary.htm and http://eprints.anu.edu.au/archive/00002196/
2. See, for example, C Edquist (ed) Systems of Innovation: Technologies, Institutions and Organizations London Pinter 1997; and C Edquist 'The Systems of Innovation Approach and Innovation Policy: An Account of the State of the Art' paper presented at DRUID Conference Aalborg 12-15 June 2001
3. See, for example, M Gibbon, H Nowotny, C Limoges, S Schwartzman, P Scott and M Trow The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies London Sage 1994; M Gibbons 'Changing Patterns of University-Industry Relations' Minerva 38 2000 pp352-361; and M Gibbons 'Innovation and the Developing System of Knowledge Production,' paper presented at Centre for Policy Research on Science and Technology at Simon Fraser University Summer Institute on Innovation Competitiveness and Sustainability in the North American Region 2001
4. See, for example, H Etzkowitz and L Leydesdorff (eds) Universities and the Global Knowledge Economy: A Triple Helix of UIniversity-Industry-Government Relations London Pinter 1997; H Etzkowitz and L Leydesdorff 'The Endless Transition: a 'Triple Helix' of University-Industry-Government Relations' Minerva 36 1998 pp203-208; H Etzkowitz and L Leydesdorff 'The Dynamics of Innovation: from National Systems and 'Mode 2' to a Triple Helix of University-Industry-Government Relations' Research Policy 29 2000 pp109-123; H Etzkowitz, A Webster, C Gebhardt, B Terra 'The Future of the University and the University of the Future Evolution of the Ivory Tower to Entrepreneurial Paradigm.' Research Policy 29 2000 pp313-330; and H Etzkowitz 'The Triple Helix of University - Industry - Government: Implications for Policy and Evaluation' Working paper 2002-11 (SISTER Stockholm) 2002
5. E McWilliam, P G Taylor, P Thomson, B Green, T Maxwell, H Wildy and D Simons Research Training in Doctoral Programs: What can be Learned from Professional Doctorates? Canberra Department of Education Science and Training 2002 p41
6. National Science Board Science and Engineering Indicators 2002 Arlington National Science Foundation 2002 (NSB-02-1) pp5-36
7. L Butler Monitoring Australia's Scientific Research Canberra Australian Academy of Science 2001
8. IP Australia
9. Major international sources include: OECD The Global Research Village: How Information and Communication Technologies Affect the Science System Paris OECD 1998; OECD University Research in Transition Paris OECD 1999; A Swan and S Brown What Authors Want: The ALPSP Research Study on the Motivations and Concerns of Contributors to Learned Journals Worthing The Association of Learned and Professional Society Publishers 1999; W S Brockman, L Neumann, C L Palmer and T J Tidline Scholarly Work in the Humanities and the Evolving Information Environment Washington Council of Library and Information Resources 2001; Institute for the Future E-Journal Usage and Scholarly Practice: An Ethnographic Perspective on the Role and Impact of E-Journal Usage among Users of Biomedical Literature Palo Alto: Stanford University Libraries and Highwire Press 2001; Institute for the Future Core Scholarly Information Tasks and E-Journal Features: Expanded Discussion Palo Alto Stanford University Libraries and Highwire Press 2002; Education for Change Researchers' Use of Libraries and other Information Sources: Current Patterns and Future Trends London Research Support Libraries Group Higher Education Funding Council for England 2002; A Friedlander Dimensions and Use of the Scholarly Information Environment Washington Digital Library Federation and Council on Library and Information Resources 2002; Key Perspectives Authors and Electronic Publishing Worthing The Association of Learned and Professional Society Publishers 2002; and D E Atkins et al. Revolutionizing Science and Engineering Through Cyberinfrastructure Arlington National Science Foundation 2003
10. National Research Council Issues for Science and Engineering Researchers in the Digital Age Washington National Academy Press 2001 p5