The Australian Library Journal

Focusing on health information: how to assess information quality on the internet

Gaby Haddow

Manuscript received November 2002

This is a refereed article

Introduction

In March 2002, the British Medical Journal (BMJ) published a collection of articles around the theme of quality assessment of health information on the internet. The number and variety of the BMJ articles reflect the amount of work being carried out in the field of health information generally, and more specifically in health information on the internet. Contributors to the issue were affiliated with organisations as diverse as the European Commission and NHS hospitals in the United Kingdom, with job descriptions ranging from 'researcher' to 'medical student'. With the exception of two authors working in the area of eHealth (potentially a link to librarianship or information management) there was no representation from our field.

However, the general literature of librarianship and information science includes material about the quality of information on the internet. Journals in our field, such as Health Information and Libraries Journal and the Journal of the American Society for Information Science and Technology (JASIS&T) carried articles in 2002 that discussed the topic. However, these publications are somewhat specialised. Health Information and Libraries Journal is aimed at a discrete audience involved in health libraries and JASIS&T is very much a research-oriented publication.

Two issues emerge: firstly, it would be useful if practitioners from all types of libraries were aware of work on the quality of information on the internet. While the information needs of library users, particularly those of public library users, vary greatly, it would be fair to assume that some of these need health information and will use the internet to find it. Awareness of the work being done in assessing the quality of information on the internet is important to ensure that the services librarians provide access to are useful and add value for their users. This is unlikely to occur if the discussion about information quality on the internet is being published in journals that few practitioners read (Ali, 1985; Haddow, 2001; Klobas, 1991). Secondly, by bringing it to the attention of practitioners and information managers it is hoped that they will consequently take an active role in the discussion about information quality on the internet; their unique perspective as organisers and facilitators of access to information will broaden the debate.

This article was written to outline the primary issues that are driving the quality discussion, to identify some of the useful tools already developed to assess quality and to illustrate, using an example of a small study, why it is important that librarians are aware of these issues.

What is 'quality'?

Any measure of information quality is attended by a myriad of individual characteristics and contexts that affect perceptions of what the term means. In traditional publishing channels, it has rarely been used to qualify information. To some extent, this is due to the publication process itself, which generally works as a filter to ensure quality (Eysenbach, 1998). For example, librarians and information managers rely upon reputable publishers, booksellers, subscription agents and reviewers who form a natural framework for providing (in most cases) credible advice to achieve collections of quality.

Information on the internet challenges these traditional measures. As any internet user knows, the webpages retrieved by a search engine can vary from highly scholarly to totally irrelevant. But in this mass there will be information relevant to a specific need, written or created by competent authors and organisations with credibility in their field. For some areas, such as information-seeking and retrieval research, the quality of information on the internet is being studied as relevance was in the past. In health science, where information quality can mean the difference between an effective treatment and a potentially dangerous and/or harmful outcome, criteria for information quality have been developed to act as a gold standard to which all health sites are encouraged to subscribe.

Quality of information on the internet: a library and information science perspective

When online information was only available via discrete databases of records, usually in a defined subject area, information retrieval researchers sought to identify the meaning of relevance to users and how systems could best meet users' requirements, this being one aspect of quality. Relevance is associated with recall and precision, and calculations rested on the approximate number of items in a database being searched and the number of relevant items in any database in relation to a specific search (Mizzaro, 1997). Clearly, calculations of this nature are not applicable when searching the internet. Furthermore, while the relevance of traditional online information derived from known sources has always been subject to variation according to individual needs, even greater variation will be seen in the assessment of information retrieved from the internet. Online information has moved out of the academy and the specialised context into the public domain.

Quality is a concept considered by Thomas Allen in the 1960s when he explored the information behaviour of research scientists and engineers (Klobas, 1995, p96). Klobas identified four components from these and other studies. They are: information quality; technical quality; relevance; and participants' interpretation of quality. The first of these, information quality, is the nearest equivalent to the way in which quality is viewed in terms of health information. It includes the attributes of accuracy; currency; authority; and novelty. The second component, technical quality, is included as a criterion for some health information assessment tools and relates to factors such as ease of navigating a web site and the use and deployment of graphics.

Quality was also a component in a number of studies conducted in the 1990s exploring users' relevance judgements (Park, 1993; Wang & Soergel, 1998; Wang & Soergel, 1999). However, it is a recent study by Rieh (2002) that brings much of this work together to focus on two factors: information quality and cognitive authority (a term first used by Wilson in 1983 to describe the judgments humans make regarding the credibility and believability of a source of information). Rieh found there were five facets of information quality: goodness; accuracy; currency; usefulness; and importance. Rieh's facets of cognitive authority are: trustworthiness; credibility; reliability; scholarliness; how official it is; and authority. These facets are similar to many of the criteria included in quality assessment tools for health information on the internet.

As noted above, many of the papers about quality of health information on the internet tend to be research-oriented: however, there is a notable exception that indicates how librarianship can contribute to the broader discussion. The DISCERN Handbook, published with funding from the British Library, was designed to be used with printed health information rather than information on the internet. The purpose of the handbook is to assist users of health information to judge 'the quality of written consumer health information on treatment choices' (Charnock, 1997). But many of DISCERN's quality criteria are relevant to both printed and electronic formats and the handbook is widely cited in health science literature.

Quality of information on the internet: a health science perspective

Much of the literature about the quality of information in health science publications has a research focus, but quality is increasingly an issue brought to the attention of all health professionals and health consumers. [See, for example, the BMJ issue referred to above and a November 2002 issue of the Weekend Australian newspaper which published a half-page article in the Health section about online health information (Gaffney, 2002)]. There is nothing new in the suggestion that some health information available in the public sphere is not accurate, is produced by people who do not have appropriate expertise, and is potentially dangerous. Previously, though, health information was not available in the quantities and with the easy access that exists now with the internet. A number of studies have been carried out to assess information about specific health topics located on the internet and all arrive at similar conclusions, epitomised by the following statement: 'there is an urgent need to check public-oriented healthcare information on the internet for accuracy, completeness, and consistency' (Impicciatore, Pandolfini, Casella, & Bonati, 1997). Accuracy is particularly difficult for health information consumers to assess because they usually have no benchmark with which to measure it.

In addition to concerns about accuracy, completeness, and consistency of health information, other issues are raised, such as currency and an absence of accountability (Bower, 1996; Eysenbach et al, 1998; Silberg, Lundberg, & Musacchio, 1997; Van Der Weyden, 1997; Wyatt, 1997). In relation to accountability, Bower (1996) noted that product advertisements on the internet can and do carry information that is in many cases based on unsubstantiated claims about treatments and cures. Moreover, the products being advertised and available through online purchase are not bound by the regulations that would apply to over-the-counter products in the purchaser's country. Silberg, Lindberg and Musacchio (1997) wrote, 'at first glance, science and snake oil may not always look all that different on the net'. These authors suggested that technology has the potential to dazzle users so that they fail to apply critical faculties they might employ in other information environments. A further aspect of accountability, not restricted to information on the internet, is an apparent reluctance of authors of health science information to declare conflict of interest, such as drug company support for their work (Tonks, 2002).

Eysenbach (1998) noted that the context in which health information appears on the internet is also important because context provides users with clues to the purpose, intended audience, and credibility of the information. Using the term 'context deficit', Eysenbach listed a number of ways that internet users could be misled by information found on or missing from web pages. One example is the lack of traditional publishing 'markers' that printed forms of information usually carry. These can indicate to readers that the information was created for a particular audience and alert them to the level of subject-specific knowledge required to fully comprehend the information. Another example of 'context deficit', noted by Eysenbach, occurs when a search engines takes a user directly to a web page without first accessing introductory information, such as a title page which may carry 'disclaimers and warnings'.

Tools to assess the quality of health information

A variety of assessment tools set out criteria which take account of the concerns discussed above. These range from simple checklists to sophisticated software programs that filter information found in an internet search (BIOME Special Advisory Group on Evaluation, 2001; Charnock, 1997; Eysenbach et al, 1998; Health On the Net Foundation, 2000; Health Summit Working Group, 1997; Pealer & Dorman, 1997). Wilson (2002) provides an excellent discussion about the different types of quality assessment tools developed and a number of these are listed at the end of this article.

The assessment tool of the Health Summit Working Group falls into Wilson's 'codes of conduct' category. Codes of conduct recommend standards that creators of websites can employ as a guide in the development phase and which consumers can use to assess the quality of the information on them (Wilson, 2002). For example, when credibility of a website is being assessed, the user is encouraged to examine the source, context, currency, relevance and/or utility, and editorial review process of the information. This process was used in a small study examining the quality of information about caring for a feverish child. This study is illustrative of many others in this context.

An example of health information quality on the internet

A total of twenty-two websites found using popular search engines (Google, AltaVista, and Yahoo!) and simple terms (child and fever), were assessed and each site was evaluated against the criteria: accuracy; currency; source; original source; and disclosure. Accuracy was determined by the extent of agreement with the findings of a systematic review conducted in 2001 which gathered evidence about the most effective treatment for a child with fever (Watts, Robertson, & Thomas, 2001). In terms of health information, systematic reviews are regarded as providing the best possible evidence about a topic. A further criterion, completeness, was applied to determine whether websites carried warnings against potentially dangerous treatments.

At best, 14 websites (sixty-four per cent) carried accurate information about how to care for a feverish child. At worst, only one website noted that sustained use of paracetamol is not advised - an important issue that has gained public exposure through the mass media in recent months (see for example The West Australian, Saturday 10 August 2002 and ABC television's 7.30 Report on 5 November 2002). The majority of websites were current, but nine (forty per cent) had no creation date. Source information was included in nearly all the websites, however, only half the websites carried disclosure information relating to the purpose and intended use of the information. Warnings about potentially dangerous treatments for a child with fever were not given by between seven to ten (thirty-two to forty-five per cent) of the websites. Overall, only three (fourteen per cent) of the twenty-two could be classed as carrying quality information based on all criteria included in the study. These findings are supported by numerous other studies (Abbott, 2000; Bidwell, 2000; Eysenbach & Kohler, 2002; Impicciatore et al., 1997; Kunst, Groot, Latthe, Latthe, & Khan, 2002; Pandolfini & Bonati, 2002).

Implications for library practitioners and information managers

The issues discussed above illustrate the challenges facing librarians and information service managers in their provision of health information to their users. Clearly, it is not enough to offer general directions to information on the internet as this can vary widely, unlike printed resources on the shelves that are more likely to have been selected within a known framework of quality judgments. These issues are particularly important for public libraries, which may be the primary source of information for some users.

There are a number of strategies that might be considered by librarians to ensure their users gain access to health information of quality.

• Reputable consumer health sites can be listed on a library's web page, providing direction for users with little prior experience.
• A short guide to criteria with which to assess the quality of information could be compiled and either given to users or left near computer terminals with internet access.
• Short information sessions about quality could be incorporated into general internet training sessions or included when users ask for advice when searching the internet.

The first list below is a collection of health information sites created by reputable organisations specifically for health consumers. A recent initiative by the Federal Government has provided all Australian internet users with free access to the Cochrane Collaboration Library, an important international source of high quality health information. Although much of the information in the Cochrane Library website is not written for consumers, the summaries available on the Cochrane website and included in the Medibank Private website are written for the health consumer audience. It should be noted that health information on the internet should not be regarded as a substitute for consulting a health professional.

Reputable websites for health consumers:

ABC Health Matters: Australian Broadcasting Corporation http://www.abc.net.au/health/

Agency for Healthcare Research and Quality http://www.ahcpr.gov/consumer/

Better Health Channel: Government of Victoria, Australia http://www.betterhealth.vic.gov.au/bhcv2/bhcsite.nsf?Open

Choice: Australian Consumers' Association http://www.choice.com.au/articles/a100481p1.htm

Cochrane Consumer Network http://www.cochraneconsumer.com

healthfinder: US Department of Health and Human Services http://www.healthfinder.gov

HealthInsite: Australian Commonwealth Department of Health and Aging http://www.healthinsite.gov.au

Health Topics A-Z: World Health Organisation - information in English, Spanish and French http://www.who.int/home-page/

Medibank Private - includes Cochrane Consumer Network information http://www.medibank.com.au/

NHS Direct: National Health Service in the United Kingdom http://www.nhsdirect.nhs.uk/

Below is a list of websites for a range of quality assessment tools developed to evaluate health information on the internet. While they have been developed for health information, many of the criteria are relevant to all types of information on the internet.

American Medical Association: Guidelines for American Medical Association websites http://www.ama_assn.org/ama/pub/category/1905.html

Discern: Quality Criteria for Consumer Health Information - a questionnaire to judge the quality of health information http://www.discern.org.uk/

eEurope: Good Practice Guidelines - developed as part of the EU Information Society program http://europa.eu.int/information_society/eeurope/ehealth/quality/index_en.htm

Health on the Net (HON): Code of Conduct - ethical standards and logo for website developers http://www.hon.ch/

Health Summit Working Party: Assessing the Quality of internet Health Information - a list of criteria for assessment of internet information http://hitiweb.mitretek.org/hswg/default.asp

HiEthics: Health internet Ethics - ethical principles for web site developers http://www.hiethics.com/Principles/index.asp

Internet Healthcare Coalition: eHealth Code of Ethics - a code of ethics for health care sites on the internet http://www.ihealthcoalition.org/ethics/ehcode.html

MedCertain: An International Trustmark for Health Information - a self- and third-party rating system for identification of trust-worthy health information http://www.medcertain.org/

Net Scoring: Criteria to Assess the Quality of Health internet Information - quality based on points awarded - to maximum 312 points http://www.churouen.fr/netscoring/netscoringeng.html

Quick: The Quality Information Checklist - designed for children and young adults http://quick.org.uk/

URAC: Health Web Site Accreditation - the logo indicates a website has been reviewed for quality http://www.urac.org/

References

Abbott, V P (2000). 'Web page quality: can we measure it and what do we find? A report of exploratory findings.' Journal of Public Health Medicine, 22 (2) 191-197.

Ali, S N (1985). 'Library science research: some results of its dissemination and utilization.' Libri, 35 (2), 151-162.

Bidwell, P (2000). 'In search of quality: the tools and techniques of health researchers on the internet'. LASIE, 31 (3), 30-41.

BIOME Special Advisory Group on Evaluation. (2001, December 18, 2001). How to evaluate and internet-based information source [WWW page]. BIOME. Retrieved 8 February 2002: http://biome.ac.uk/guidelines/eval/howto.html

Bower, H (1996). 'Internet sees growth of unverified health claims.' British Medical Journal, 313 (7054), 381.

Charnock, D (1997). The Discern handbook: Quality criteria for consumer health information on treatment choices. Oxford: Radcliffe Medical Press.

Eysenbach, G, Diepgen, T L, Muir Gray, J A, Bonati, M, Impicciatore, P, Pandolfini, C, & Arunachalam, S (1998). 'Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information.' British Medical Journal, 317 (7171), 1496-1502.

Eysenbach, G, & Kohler, C (2002). 'How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews.' British Medical Journal, 324 (9 March), 573-577.

Gaffney, D (2002). 'The good and bad of online health care.' The Weekend Australian, Health section, November 23-24, 4.

Haddow, G C (2001). The diffusion of information retrieval research within librarianship: a communication framework. Unpublished Doctoral Thesis, University of Western Australia, Perth, Western Australia.

Health On the Net Foundation. (2000, November 20, 2000). HON's Initiatives and Services: MARVIN [WWW page]. HON. Retrieved 8 February 2002, from the World Wide Web: http://www.hon.ch/Project/Marvin_project.html

Health Summit Working Group. (1997). Criteria for assessing the quality of health information on the internet: Mitretek Systems and AHCPR.

Impicciatore, P, Pandolfini, C, Casella, N, & Bonati, M (1997). 'Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home.' British Medical Journal, 314 (7098), 1875-1881.

Klobas, J E (1991). 'Finding out about developments in information technology: A comparison of the information-seeking behaviour of librarians and computing professionals.' Online Information 91 (10-12 December 1991. 379-391.

Klobas, J E (1995). 'Beyond information quality: fitness for purpose and electronic information resource use'. Journal of Information Science, 21 (2), 95-114.

Kunst, H, Groot, D, Latthe, P M, Latthe, M, & Khan, K. S. (2002). 'Accuracy of information on apparently credible websites: survey of five common health topics'. British Medical Journal, 324 (9 March), 581-582.

Mizzaro, S (1997). 'Relevance: the whole history'. Journal of the American Society for Information Science, 48 (9), 810-832.

Pandolfini, C, & Bonati, M (2002). 'Follow-up of quality of public oriented health information on the world wide web: systematic re-evaluation.' British Medical Journal, 324 (9 March), 582-583.

Park, T K (1993). 'The nature of relevance in information retrieval: an empirical study.' Library Quarterly, 63 (3), 318-351.

Pealer, L N, & Dorman, S M (1997). 'Evaluating health-related web sites.' Journal of School Health, 67 (6), 232-235.

Rieh, S Y (2002). 'Judgment of information quality and cognitive authority in the web.' Journal of the American Society for Information Science and Technology, 53 (2), 145-161.

Silberg, W M, Lundberg, G D, & Musacchio, R A (1997). 'Assessing, controlling and assuring the quality of medical information on the internet: caveat lector et viewer - let the reader and viewer beware.' JAMA, The Journal of the American Medical Association, 277 (15), 1244-1246.

Tonks, A (2002). 'Authors of guidelines have strong links with drugs industry.' British Medical Journal, 324 (16 February), 383.

Van Der Weyden, M B (1997). 'Medical information and the World Wide Web.' MJA, 167, 571-572.

Wang, P, & Soergel, D (1998). 'A cognitive model of document use during a research project. Study I. Document selection.' Journal of the American Society for Information Science, 49 (2), 115-133.

Wang, P, & Soergel, D (1999). 'A cognitive model of document use during a research project. Study II. Decision at the reading and citing stages.' Journal of the American Society for Information Science, 50 (2), 98-114.

Watson, R (2002). 'European Commission to publish a code of practice for websites.' British Medical Journal, 324 (9 March), 567.

Watts, R, Robertson, J, & Thomas, G (2001). The nursing management of fever in children (Vol 14). Adelaide: The Joanna Briggs Institute for Evidence-Based Nursing and Midwifery.

Wilson, P (2002). 'How to find the good and avoid the bad or ugly: A short guide to tools for rating quality of health information on the internet.' British Medical Journal, 324 (9 March), 598-600.

Wyatt, J C (1997). 'Commentary: measuring quality and impact of the world wide web.' British Medical Journal, 314 (7098), 1879.

Biographical information

The author is currently working as a Research Project Officer for the WA Centre for Evidence-based Nursing and Midwifery, a collaborating centre of the Joanna Briggs Institute. Her background is in librarianship, with a special interest in the relationship between research and practice. G.Haddow@curtin.edu.au