AARL

AskNow! - Evaluating an Australian collaborative chat reference service: a project manager's perspective

Fran Wilson and Jacki Keys

Abstract
AskNow! online answers Australia-wide, the first nation-wide chat reference service, has been a successful exercise in collaboration and cooperation.
This article examines some of the lessons learnt, the challenges of collaboration and possible future directions. AskNow!, an initiative of the Council of Australian State Libraries (CASL), commenced as a twelve-month pilot using 24/7 Ref software in August 2002, staffed by national, state and territory libraries.
Over 40 000 enquiries, exit surveys and system reports provided client demographics and user satisfaction ratings indicating AskNow! was well received by regional and urban Australians. In March 2004, CASL endorsed AskNow! as an ongoing service.

The advent of the internet, innovative technology and the increased availability of electronic resources have impacted dramatically and globally on the provision of reference and information services. As the number of people 'living online' grows, and as face to face reference declines, libraries have responded by creating virtual reference services to assist all users, both onsite and remote, breaking down barriers to information access. Collaborative reference services - local, regional, national and international - have been established in many countries including the United States, Sweden, Denmark, Norway, Germany, Canada and Australia. Chat or live reference is a significant feature of many of these collaborations.

Key features of chat reference services include the ability to interact in real time with a patron, regardless of location, to answer reference questions, to use scripted messages, to 'push' websites and to 'co-browse' - that is, librarian and patron can simultaneously view the same website. At the end of the chat interaction, patrons who have provided an e-mail address in initial connection to the service, receive a session transcript including links to the websites suggested. Operators are also able to help several patrons concurrently and some software also has the option of screen sharing where the screen of the user can be viewed by the librarian. An added advantage of the co-browsing and screen sharing features is the opportunity they offer a librarian to teach users to become more information literate and self sufficient.

The Council of Australian State Libraries (CASL) established the Reference Issues Working Group in early 2000 to explore reference trends and to investigate the potential for collaboration in the delivery of reference services. Each of the nine libraries - national, state and territory - was already offering a range of services to remote library patrons, including telephone and e-mail reference and the provision of web enquiry forms. Some libraries had started to investigate chat reference and were testing selected software, but had not committed funds to purchase any particular product.

At a face to face meeting of reference managers from CASL libraries in late 2001, it was unanimously decided to trial a collaborative model using chat reference software. A business plan submitted to CASL in early 2002, proposing a 12 month pilot of a collaborative chat reference service, was endorsed and the planning began.

Project development

The first stage of the project involved establishing a team with representatives from each state and territory library. The National Library provided project management. The participating libraries were the State Library of NSW, State Library of Victoria, State Library of Queensland, State Library of South Australia, State Library of Tasmania, State Library of Western Australia, the Northern Territory Library and Information Service, the ACT Library and Information Service and the National Library.

The overall objectives of the project were to:

• define and test the collaborative service model
• test and evaluate selected interactive chat software applications
• evaluate and assess customer needs and market segments for online reference
• test and evaluate resourcing required to deliver online reference, and
• deliver a collaborative reference service.

A request for quotation was sent to a shortlist of vendors. The software was tested over several weeks in each of the libraries against the selection criteria, and 24/7 reference software from the Metropolitan Cooperative Library System (MCLS) and hosted in the United States, was selected. Initially a licence for two seats was purchased allowing two librarians to operate the service concurrently. A deed of agreement - CASL Live Reference Deed - established the consortium, outlined the management of the pilot, including the responsibilities of participating members and the National Library and the procedures applying to the negotiation and execution of the licence for the service. The service was not expected to replace other reference and information services but to supplement them, offering library patrons an additional means of communication with libraries. It was decided that questions received, if not answered immediately, would be referred on to the relevant e-mail service wherever possible, after negotiation with the enquirer. This eliminated the need for the patron to ask the question again through e-mail.

The deed detailed the division of costs using a three-tier system determined by the size and budget of each participating member library and it was expected that the larger libraries in the consortium would staff the service for more hours than the smaller libraries.

Project development and communication was through regular teleconferences and via an electronic mailing list. After the name of the service had been chosen, AskNow! online answers Australia-wide, an application to the National Office for the Information Economy (NOIE) for the domain name successfully established the pilot with a distinctive URL - http://www.asknow.gov.au

The AskNow! website, along with the logo, was designed and developed in-house at the National Library, in consultation with the project team. The service's website comprised two parts, one with information about AskNow! taking up two thirds of the web page and customisable. The remaining third comprised the name, postcode, e-mail and question fields, also customisable, which was developed in liaison with MCLS.

A password-protected administrative website, hosted at the National Library, facilitated sharing of project documentation such as the deed, the project plan, the roster and service guidelines, statistics, survey results and key websites. Session transcripts were accessible via a password protected section of the 24/7 reference site which also enabled the generation of a range of reports by day, week or month and the facility for creating scripted messages, amending library details, logins and passwords.

Training and implementation

The project team developed service guidelines to provide direction, standards and consistency. These included scheduling and staffing, emergency procedures, optimum length of reference transactions, responses to clients, managing technical difficulties and dealing with inappropriate enquiries and the procedures for the provision of copies and integration with other services. They were amended, when necessary, as the pilot progressed.

Training was undertaken in the weeks leading up to the go live date with each team member responsible for training in their state. Each state had unique logins and passwords allowing them to train at times convenient to them. In most cases, staff would alternate between acting as librarians or clients to test and explore the functionality of the software. A training CD-ROM provided to each state as part of the contract was not as useful as the hands-on experience. Most states opted to train a small group of staff in the first instance, as it was agreed that not all librarians would be suited to this type of reference interaction. It became apparent that a certain set of skills was required to operate effectively as a chat librarian - internet search abilities, technological savvy, lateral thinking and quick fingers - 'Librarians who cannot multitask well or do not enjoy multitasking should probably not be involved in providing this service'. [1]

AskNow! commenced in August 2002 operating for 50 hours per week from 9:00 am to 7:00 pm Australian Eastern Standard Time. The logo appeared on the websites of each participating library, with distinctive HTML code behind each, enabling the librarian to see from which library the user was accessing the service. The website for that library would show as the referring URL when the librarian connected with the user, enabling better service, more focussed on their specific information needs. There was limited marketing and promotion in the early stages of the pilot. The major emphasis in the first four months was on staff training and bedding down the new service. In addition to the exit survey, questions and comments were also received through an AskNow! e-mail address which was monitored daily by staff at the National Library.

The official launch in Sydney and related media publicity in January 2003 marked the second stage of the pilot resulting in a huge upsurge in demand and the purchase of a third seat. In May, ABC radio interviews broadcast across several states promoted the service to regional Australians. The final stage of the pilot involved, at various stages, the participation of selected public libraries and the National Library of New Zealand and the National Library Board of Singapore which tested the potential for cooperation with our regional partners. While the two national libraries did not make the AskNow! logo available on their websites, they requested participation in the pilot in order to explore the delivery of reference services through a new medium. They provided a commitment of hours per week in staffing the service.

Evaluation of the pilot

The pilot was initially approved for 12 months and an evaluation was undertaken at the mid term point. Regular reporting to CASL detailed the progress and usage of the service and highlighted any operational or strategic issues. At the end of the 12 month period, CASL approved an extension of the pilot to the end of 2003, to allow a full evaluation which would inform the viability of an ongoing service. CASL subsequently approved a further extension to the end of March 2004.

The formal evaluation of AskNow! was conducted by the State Library of NSW in consultation with the member libraries and the project's steering committee, CASL's Reference Issues Working Group. It is worth noting that continuous informal assessment and monitoring of the service, including the customer feedback and the transcripts, enabled the service to be modified and enhanced throughout the pilot period. Such ongoing analysis allowed the team to survey usage patterns, types of questions and respond to comments received.

Methodology

Three data gathering methods were used for the evaluation:

• statistics and transcripts from the service provider
• an exit survey for users, and
• a survey of operators (focus group and questionnaire).

In considering the demographic profile of AskNow! users, data from the Australian Bureau of Statistics (ABS) was used as a comparison. [2]

The evaluation covered the period September 2002 to November 2003. For the transcript analysis and exit survey analysis, two months were chosen as representative of the evaluation period, October 2002 and May 2003.

Usage and capacity

Over 40 000 enquiries were received during the pilot (see Table 1). Of these enquiries, 74 per cent were completed during the chat session with 6 per cent referred on to the e-mail services of the participating libraries.

Table 1
Total number of enquiries received

Using data from the two sample months, the evaluation revealed that the duration of chat sessions ranged from one minute to 53 minutes with eight minutes the average; 70 per cent of clients waited one minute or less from the time they initiated the session before receiving a response. With three seats operating simultaneously, 23 enquiries could be completed in an hour. However, on particularly busy days, the service handled well in excess of 23 enquiries an hour, reaching into the high 60s. This was demonstrated by the percentage of enquiries that were abandoned - 18 per cent - when the patron logged off before a librarian responded. The service was more popular than anticipated and at times could not keep pace with the demand.

Who used the service?

The evaluation revealed that when compared to ABS population statistics,3 clients under 24 years of age were over represented by about 15 per cent, and clients over 60 years under represented by about 13 per cent. This does not seem unexpected given the fact that 84 per cent of Australians aged between the ages of 18-24 use the internet. [4] All other age groups were roughly in line with the population breakdown by age. Clients under 16 years of age made up 29 per cent of all users, demonstrating that the service appealed particularly to younger clients.

Analysis of the session transcripts revealed that there were four main user groups - the under 16 years, the general users which broadly included members of the public, college and tertiary students, government employees, family historians and librarians asking on behalf of clients, advanced researchers and users specifically asking about the services, facilities or collections of the participating libraries.

Table 2
AskNow! client age and ABS comparisons

Sample size n = 908
Source: AskNow! survey data, ABS 3201.0

Where were they from?

Most clients were from New South Wales or Victoria, again in line with ABS population statistics. [5] There was some under-representation of Queensland clients, while ACT residents represented a higher than average number of clients who were 35 years and older. Overseas clients made up just 2 per cent of the total.

Table 3
AskNow! client and general population

Sample size n = 908
Source: AskNow! survey data, ABS 3101.0

Regional Australia was well represented. Clients from country areas made up 30 per cent of users which closely reflects the proportion of Australians from these areas. These clients are less able to access traditional library services of the state and territory libraries located in metropolitan areas. Comments received from these users affirmed this.

This is extremely useful for rural people. We are a rural law firm with limited access to research materials locally outside of our law library.

The majority of users - 61 per cent - went directly to the AskNow! website rather than accessing it through the websites of the individual libraries.

What did they ask?

The analysis of the session transcripts over the two sample months confirmed that questions received through AskNow! covered a wide range of subjects, ranging from the very simple to complex research. History and Australian history together made up 11 per cent of all enquiries, with applied science and technology following closely with 10 per cent and science with 9 per cent.

Queries about specific libraries and their collections or services comprised between 5-10 per cent of the total received. Other notable subject areas included health, performing arts, geography and travel and family history. Information available on the internet was used to answer 58 per cent of enquiries.

Research enquiries and enquiries which required the librarian to check print resources or were specific to a particular library's services where information was not accessible online, were generally referred to the relevant participating libraries' e-mail reference services. This procedure was in accordance with the service guidelines developed at the beginning of the pilot.

What did they think of the service?

The exit survey asked respondents to rate the overall quality of the service, the information provided and the service's ease of use. Respondents were also asked if they would use the service again. A comments box yielded useful feedback and suggestions for service enhancement.

Analysis of the exit surveys and session transcripts indicated a high level of satisfaction with AskNow! In particular, users commented on the fast response, the quality of the information received and the helpful staff. Comments were also received that endorsed the concept of a chat reference service. According to the exit survey sample, 70 per cent of respondents rated AskNow! as excellent or very good, and 91 per cent said they would use it again. This was backed up by comments such as:

Very happy, it's great to be able to secure information from another state, without having to physically go there in person.

I have been looking on government websites for ages and could not find what I wanted and you gave me the information straight away.

This is a wonderful reference service. I appreciate the logic in the design and innovations such as this highlight the value of professional library staff to the community.

Really great idea. Hope there's lots of librarians there when the news gets out!

This tool is great for deaf people as we tend to avoid asking questions because we feel we are slowing down others at the desk.

This last comment illustrates that the service has the potential for better meeting the needs of some clients with disabilities.

Clients expressed dissatisfaction however when they experienced technical difficulties, or when kept waiting for long periods when the service was very busy. Despite increasing the number of seats to three, demand still exceeded capacity. This increased pressure on staff. Client expectations of the service also generated dissatisfaction. Many clients expected a near instant response and were not aware of the time needed to perform a reference interview in order to fully explore the question being asked. The client experience was also less successful when assumptions were made by the operator about the client, eg that they had not already searched for the information sought.

... I also disliked that the Librarian presumed that I hadn't looked for the info on the internet first and consequently pushed thru web pages that I had already found myself. Perhaps you need to establish the research skills of your user first.

The nature of the collaboration also caused confusion with clients who were intentionally attempting to contact a specific library -

... why allow Victorian users (who want answers from the SLV in Victoria) to access a system that goes to a librarian in Tasmania who then refers it to the State Library of Victoria... doesn't make sense!

It is obvious from comments like this that the 'call centre' nature of the service needed to be better explained through appropriate marketing and promotion and that suitable methods for dealing with enquiries specific to one of the libraries, further explored.

What did the librarians think?

From the start of the pilot, library staff accepted and welcomed the challenge of delivering reference services using chat technology. From the operator survey (sample of 94), 85 per cent agreed that AskNow! was a valuable information service and 75 per cent agreed that it had increased their web searching skills. Operators were, however, concerned with the level of inappropriate behaviour, such as abuse from school students and 95 per cent agreed that more work was needed to ensure students use the service in a more appropriate way. There was also concern about the resourcing implications, with 40 per cent of staff disagreeing that there were sufficient staff to support the service.

AskNow! operators also supported the importance of improved marketing with 90 per cent strongly agreeing or agreeing that the service needed to target specific groups such as regional Australians. There were also concerns with the stability of the software with 62 per cent of respondents rating the software as fair or poor. It is worth noting here that it was often very difficult to assess where the problem lay when technical difficulties were experienced, as there are so many factors involved in a chat interaction. The software does not operate in isolation but is intricately linked to factors such as the amount of internet traffic, the particular library's network, the speed of the patron's connection, the power of their PC, the type of browser and the location of the server hosting the service.

Operators generally considered that the service guidelines were appropriate but stressed the need for all librarians to adhere to the guidelines in order to maintain a consistent level of service.

Lessons learnt

Two main issues became apparent during the AskNow! pilot - the high level of usage by school students and the technical problems associated with a chat reference environment.

The resolution of the technical issues associated with the software and the internet will improve in time as the software becomes more sophisticated and more stable and bandwidth for internet access improves. However, the high level of usage by school students will be a more difficult issue to resolve. Inappropriate usage of the service by students is not a problem unique to AskNow! There is regular discussion on electronic mailing lists, such as the 24/7 reference forum, highlighting rude student behaviour as a major concern and librarians communicating via mailing lists have worked in collaboration to devise suitable scripted messages. Janes states that 'synchronous technologies may appeal to groups of users... particularly the young, who are addicted to the social nature of instant messaging and chat technologies'. [6] This was reflected in the finding that 29 per cent of our users were aged under 16 years. Another area of concern was the usage of AskNow! by classes of students, all asking the same question.

The project team brainstormed strategies to deal with the high usage by school students, including class groups. One of these was to develop a link from the service page to tips for students - which listed what the service could and could not do for them. It included some hints on effective web searching using Google and selected homework help sites. The availability of this link appeared to have minimal impact. Once users found the dialogue box, it was too late to channel them to other sites as they were focussed on typing in their question and connecting to a librarian. Many other services now provide an intermediate page outlining what customers can expect from the service and providing links to homework help. Of course there is no guarantee that users will read this page either, but it is worth trying.

Janes also highlights another problem regularly experienced by many chat services, that of users disconnecting or not answering - 'users simply evaporate in the middle of the chat session. Have they found the answer, gotten bored and logged off, or was their connection severed? [7] 'This was particularly frustrating when the user had not provided an e-mail address thereby guaranteeing the delivery of the session transcript afterwards. Despite the message at the time of connection espousing the benefits of entering an e-mail address, many ignored this advice. Of course some clients may not have had access to an e-mail account. Librarians were left wondering whether the user had received the information they had provided. Developments in software, including 24/7 reference, now provide the option of anonymous login. A link to the transcript is made available after the user logs off.

The pilot highlighted both the opportunities provided by collaboration and the challenges. No library could have offered this service alone and the advantage of sharing the number of hours and the existence of three different time zones in Australia (five during daylight saving) meant that the service was available for longer hours each day. Further, the collaborative model facilitated an increased knowledge and awareness among the participants of each library's collections, services and facilities. However, the collaborative model certainly highlighted the technical problems. This is almost inevitable when there is a variety of hardware configurations, networks, operating systems, proxy servers and varying levels of IT support and assistance. The issues surrounding the use of licensed databases also requires consideration when operating a reference service collaboratively and strongly supports the case for national site licenses for core electronic data sets.

AskNow's client base was not a defined group such as a university or the residents of specific town or region who have to provide a means of authentication in order to use the service, but was made accessible to all. This led to an influx of new users, many of whom were not traditional users of the participating libraries.

It is fair to question the quality of reference delivered through a chat service, in comparison to other modes of reference, particularly when you limit the amount of time spent with each customer. Our service guidelines stated that each operator should spend no more than 15 minutes with each user. This was often very difficult to do. Chat reference is undoubtedly more labour intensive than telephone or face to face reference, due to the time required to send and receive messages. The demand for AskNow! certainly necessitated a time limit, but is this always appropriate? '... many libraries are restricting their chat-reference questions to those with short answers... will this stifle the potential for online chat-reference services?' [8] However, the unique advantage of chat reference is that, depending on the multi-tasking skills of the librarian, several customers can be juggled simultaneously.

The branding of the service could also have resulted in client expectations not being met - 'What we call things matters'. [9] The subtitle to the service 'online answers Australia-wide' suggested that clients using this service would be provided with answers. In many cases, particularly with students, operators were teaching them how best to use the internet to find relevant information and/or pointing them to other sources, both print and electronic, where the answer could be found. It may have been better branded 'online information' or 'online reference help'.

The timing of the launch of the service and the subsequent marketing in retrospect unwittingly geared it to the school market, occurring as it did, at the beginning of the school year. Questions and feedback received in the first four months of the service indicated that there was real potential for servicing clients of all ages. It was obvious from the surge in traffic that the message clearly tapped into the school market to the possible disadvantage of other users.

The wide variety of skill levels including reference experience and knowledge of information sources among the librarians staffing the service, combined with different approaches to dealing with clients, made quality control in a collaborative framework a difficult and potentially very sensitive issue. The availability of recorded transcripts enabled librarians to review the quality of the service and the answers. This provided opportunities to address the issues of quality and consistency, but also opportunities for staff development, training and information sharing.

Conclusion

In mid March 2004 at the meeting of the Council of Australian State Libraries, the business plan, proposing the establishment of AskNow! as an ongoing collaborative service, was endorsed.

The pilot certainly achieved its objectives to ascertain the market segments, demand for and resource requirements of a collaborative chat reference service. Judging by the high demand for this service from school students, liaison with the education sector is imperative. The Reference Issues Working Group and the CASL Partnerships between Libraries and Education Working Group will work together in liaising with the education sector to explore virtual reference services for schools. An initiative of the National Library of New Zealand is currently exploring a chat reference service geared directly at school students. This initiative will be closely monitored and the possibility of collaboration between Australia and New Zealand in servicing the school market warrants further investigation.

As the service evolves alternative models may be considered, such as an extended collaboration involving numerous sectors - school, public and university libraries, state, national and territory libraries or the development of separate services catering to different client groups. Chat services in the United States such as MyWebLibrarian funded by the Illinois State Library are now providing options for clients to select their relevant group or information need, eg general audience, college students and faculty and health and medical information, routing them to the most appropriate service from the one central point. Consolidation of partnerships with libraries from the Asia-Pacific region is also possible.

The AskNow! pilot afforded staff at the national, state and territory libraries in Australia the opportunity to explore the potential for collaboration using the latest technology for delivering reference services - chat software. A dedicated and committed project team inspired enthusiasm in their respective teams to embrace this innovative and challenging mode of interaction with their clients. The AskNow! team and all the operators have learnt valuable lessons along the way.

We are now in a much better position to know what it is we want and expect from a chat reference service: better queue management and reporting, facilitated transfer of patrons from one operator to another, more stable software and timely support. We are aware of the need to carefully identify and segment the different client groups, to specify exactly what the service will and won't provide and to plan and deliver appropriate and targeted promotion and marketing. We also want the ability to more easily create a knowledge base using questions and answers that are embedded in the individual transcripts. We want the facility to develop, modify and access a list of FAQs and the option, where appropriate, to reuse an answer to a question that has just been asked. We expect software providers to provide blocking of IP addresses as a mechanism for dealing with repeat rude and offensive clients. Voice over IP, where the librarian can talk to the patron at the same time as co-browsing websites, and digital video, have the potential to further enhance service and their development and suitability will be monitored and carefully assessed.

Librarians have embraced and applied information technology to enhance and extend services. Chat reference is the latest technological innovation to impact on reference services. AskNow! has been a successful pilot. It has confirmed that the collaborative model can and does work. There is now a need to consolidate our knowledge, focus on the future and the further development of suitable services for different market segments to ensure the library experience is a positive and enriching one for all.

Notes

1. D Helfer 'Virtual Reference in Libraries: Status and Issues' Searcher p3 vol 11 issue 2 February 2003
2. Australian Bureau of Statistics Australian Demographic Statistics 3101.0 2002
3. Australian Bureau of Statistics Population by Age and Sex 3201.0 2003
4. Australian Bureau of Statistics Household use of Information Technology 2001-2002, 8416.0 2002
5. Australian Bureau of Statistics Australian Demographic Statistics 3101.0 2002
6. J Janes 'Live Reference: Too Much, Too Fast?' netconnect p13 Fall 2002
7. Ibid
8. C M Johnson 'Online Chat Reference : Survey Results from Affiliates of Two Universities' Reference & User Services pp237+ vol 43 issue 3 March 2004
9. A Pace 'Virtual Reference: What's in a Name' Computers in Libraries p56 vol 23 issue 4 April 2003

Recognition of AskNow!

The AskNow! pilot service was recognised by a 2003 Arts Portfolio Leadership award in Victoria (leadership in customer service/marketing/audience development category). It is also featured, as one of 20 case studies highlighting achievements in e-government, in the National Office for Information Economy (NOIE) publication Transforming government: achievements in e-government, Canberra, 2003.

AskNow! was also named in the Senate Environment, Communications, Information Technology and the Arts References Committee's report Libraries in the online environment, Canberra, October 2003.

Fran Wilson (please remove '.nospam' from address), AskNow project manager March 2002-November 2003. Manager, information services, National Library of Australia.

Jacki Keys (please remove '.nospam' from address), AskNow project manager November 2003-March 2004. National Library of Australia.