 home > publishing > aarl > 32.2 > full.text > article |
|||
Analysing the Quality Gap:Reflections on Results from an Australasian Universitas 21 Libraries Standard Survey of Service QualityIsabella Trahn, Jennifer Croud, Karen Kealy, and John Hayward AbstractIn 1999 four large Australasian university libraries undertook a common client perception of service quality survey based on aspects of SERVQUAL concepts. The survey measured client perceptions of the service expectations of more than 40 items describing aspects of service provision, client perceptions of performance on these items and hence the perceived gaps in service provision. Brief conclusions are drawn on client characteristics, responses and areas in which strategies are being implemented to target particular service quality improvement priorities. The participating libraries are members of the Universitas 21 Libraries group. Since reports on orchestrated survey projects across a number of institutions are not yet common, this information may be of wider interest and may also contribute in a small way to the library service quality research agenda which aims to 'Develop customer-based measures that complement or replace performance input and output measures'.1 Universitas 21 is a network of 18 universities in 10 countries. It enrols about 500 000 students annually, employs some 44 000 academics and researchers, and has a combined operating budget of almost $US9 billion'.2 The Libraries Group is an association of kindred university libraries, all part of U21 institutions. Goals include to:3
The libraries of the four Australasian universities (Melbourne, New South Wales, Queensland and Auckland) have formed an Australasian Libraries Group within the broader U21 Libraries grouping. Using a SERVQUAL - Based InstrumentIn late 1999 there was a discussion of first results from the co-ordinated use of an essentially common client perception of service quality survey instrument to measure service quality. The source survey was based on modified Parasuraman SERVQUAL principles.4 The group decided on absolutely minimal changes to this 1997 Melbourne survey in order to retain the validity and consistency of the instrument. The naming of the dimensions of service had already been modified for the Australian service industry and library context before the source instrument had been used in 1997 in Melbourne. Other researchers and projects have similarly amended the service dimensions as the table below indicates.
Debate continues on the validity and construction of different dimensions of library service quality; however, another, larger existing difference was the restriction of client Likert scale responses to two (perceptions of service expectations and perceptions of performance) for each service issue rather than Parasuraman's three (minimum expectation, desired expectation and perception of library performance) which are still contained in the LibQUAL+ project and a recent Singaporean study.9 A joint summary and analysis of results was produced by a small working group as applied research which could benefit the participant libraries. Until the full results of the US Association of Research Libraries large scale LIBQUAL+ project10 become available there are few reported examples of the orchestrated use of standard survey instruments by groupings of large research libraries, so information from this Australasian work may be of broader interest. The ARL pilot survey, based on SERVQUAL concepts was carried out in 12 large US research libraries in April/May 2000.11 Initial results were released at the October 2000 Washington conference 'New Culture of Assessment in Academic Libraries: Measuring Service Quality.12 A revised instrument is being administered in a larger number of North American research libraries in spring 2001. The Australasian experience predated the US project and involved four similarly sized academic libraries with reasonably similar discipline coverage. Brief conclusions are drawn on the characteristics of the clients, their responses to their own library experience and, importantly, the areas in which strategies are being developed using the data to direct relevant service improvement processes. The key outcome for this exercise was, and is, service quality improvement. Despite technical inadequacies in this first co-operative survey, useful data were provided to promote best practice service delivery. Brief Library SnapshotsUniversity of Melbourne
23 branch libraries University of New South Wales
highly centralised service University of Queensland
13 branches on the St Lucia, Gatton and Ipswich campuses, in teaching hospitals and at the Dental School University of Auckland
three campuses 1999 SurveysEarly in 1999 the U21 Australasian Libraries Group decided to run a common client perception of service quality survey very little altered from one developed for Melbourne in 1997. For one example of the actual survey instrument see the Queensland website HREF=http://www.library.uq.edu.au/about/CustomerSurvey99. pdf.13 The raw results from all four libraries were added to a benchmarking database at the Victorian-based Rodski Behavioural Research Group, and compared with data from a variety of other organizations around Australia. At the time, the perceived benefit of running a survey across all sites was that the group would be able to compare results for benchmarking, and be able to share service quality improvement ideas. University of MelbourneIn 1997 the dimensions (or actionable variables) for grouping the 40 plus original survey questions were selected on the basis of establishing a perception of service quality for each and enabling results to be benchmarked in the Rodski database against other organisations using similar dimensions. As the library had already received an Australian Quality Award for Business Excellence it was critical that there was alignment between the Rodski benchmarking and the Australian Quality Council's Business Excellence Framewor k (ABEF). The ABEF is used to measure and improve current performance. The use of a bi-variate scale also enabled areas of large service expectations/performance gaps to be identified. The 1997 survey was modified and run again in 1999, in both paper and web versions. University of New South WalesThe original aim of the survey was to gauge the opinions of the major clientele on how well they thought the library performed in relation to the perceived service expectations of items, in order to focus on areas where gaps were largest. A modified 1997 Melbourne version had been run in 1998, modified again in a minimal way and conducted at the same time in second session 1999. Again, survey data were sent for benchmarking against other library data and data from the broader group of organisations in the Rodski database. UNSW also makes use of the ABEF framework to improve library services. University of QueenslandThe library ran a slightly modified Melbourne survey for the first time in 1999. Aims were to identify key customer concerns; manage key prioritised issues affecting customers; provide the ability to measure and monitor performance over time; allow customers to communicate openly and honestly with the library; benchmark results against Australasian Universitas 21 partner libraries and other organisations in the Rodski database. The survey was offered in both paper and web versions. Data were analysed and benchmarked by Rodski. The alignment between the Rodski benchmarking and the ABEF was considered useful as the library also is using the framework to measure current performance and put critical issues into a library-wide context. The library will repeat the survey in 2001. University of AucklandThe library saw benefits in conducting a survey using a shared instrument and made very few modifications to Queensland's version. This was also seen as a way to maximise the comparison of results for benchmarking purposes. The library conducted its first survey in 1999 and intends to repeat it biennially. The decision to administer the survey as a web version only was influenced by timing. It took three weeks from planning to implementation. This was important, as an adequate sample size of responses was needed before the onset of end of year pressures. Survey objectives matched Queensland's. Using the web meant that results could be analysed immediately and the report produced. Survey Process, Response Demographics and Frequency of Library UseIndividual variations in the survey process and response demographics are detailed below. A focus of discussion was the question of individual institutional choices of vehicle for the survey (paper only, mixed paper and web and web only) and the consequences for survey results. There is much to investigate further regarding the conduct of surveys in different media. Questions of equity, sampling, technical administration, length, ability to complete correctly, cost and other aspects will be pursued prior to any further iteration. There are many unresolved issues of more than academic interest. Discussion then flowed to size differences, disciplinary characteristics and variations in the patterns of respondent populations. The number of respondents involved varied from around 300 to over 1000. It is interesting to note that the Nitecki reports of a Yale survey sourced only 226 responses,14 the Texas A&M report was derived from small sized yearly samples also (average of 266 responses per year over three years)15 as was the more recent Tan survey in Singapore (66 responses).16 The proportional representation of the client populations - undergraduate students, postgraduate students, staff and other were:
Nitecki's Yale survey respondents included 16.5% faculty and 'staff' with most of the remainder being students.17 There were varied speculations on the reasons for differences. Differences in delivery mechanisms may have been an issue. Faculties with the largest numbers of responses were, in descending order, each institution having 7 to 10 faculties:
There was no further comparison, at this point, of this pattern with faculty size in the individual institutions. Two of the four libraries also collected data on frequency of use of the physical library facilities. University of MelbourneAdministered in October 1999, the survey was on the library web site and paper copies were available in all libraries for equity reasons, for those who were unable to access the web or who preferred a paper copy. The response rate was dramatically reduced from previous years. Variables which may have contributed to the low response rate include time of year, first web version, survey length and survey overload as there are many university surveys.
University of New South WalesResponse statistics were as follows:
The response to the standard mailout based on random sampling (sampling 1 in 5 postgraduates; 1 in 10 undergraduates) was characterised by a high postgraduate return. The undergraduate response was proportionately lower than in the other libraries. The disciplines differed slightly from those at the other institutions. International students were 27% (206) of UNSW respondents, in keeping with the nature of the UNSW community. There was no significant difference between international and other students in the average service expectations ratings of the ten most important items. At UNSW the ratio of daily, weekly and monthly visits was 31:52:17. A US pattern reported at Yale was 19:64:17. At UQ the figure was 51:40:8. The higher proportion of academic staff and postgraduates in the UNSW sample may be a factor in proportionately fewer daily visits than UQ. Daily visits would appear to be more common in Australasia than in US institutions such as Yale. For UNSW international students the figures for frequency of visits were 41:44:8. As has been noted in other surveys, international students tend to visit the library more frequently than other students. University of QueenslandThe survey ran over four weeks in August/September 1999 under the title: 'How well you think the Library performs in relation to what you think is important?' Paper copies were posted to 3500 randomly selected staff and students and were available at branch library information desks. The web survey was made available on the library web site, linked from the home page. User opinion on 43 variables was sought in the dimensions of Communication, Service Quality, Service Delivery, Facilities and Equipment, and Library Staff. Response statistics were as follows:
8.8% of respondents indicated they were international students. Approximately 10% of the student population are international students. The ratio of daily, weekly and monthly visits to the Library was 51:41: 8. University of AucklandThe survey ran for three weeks from late September 1999. It was conducted entirely electronically an d accessed through the library's LEARN website available from within the library system, elsewhere on campus workstations, and remotely. A prize was offered. Library staff encouraged people to enter through e-mail lists, faculty meetings, workstation posters, notices, etc. At the aggregate level the 1229 (932 usable) responses provided a high degree of confidence in the results obtained through quantitative analysis. Results for some specific demographic categories were low and thus interpreted with caution. Overall Perception of Service QualityFigures for the extremes of perception of high service quality (6 or 7 out of 7 = excellent) and perception of low service quality (1 or 2 out of 7 = poor) were:
Post survey U21 discussion focussed on specific areas of service and the response to individual items rather than on these overall perception of service quality ratings. University of MelbourneThe mean overall perception of service quality rate across the whole library system was 4.7. Although there were some areas of frustration or perceptions of lower quality service in relation to some libraries by each category of customer, there was a large degree of similarity in the issues identified across the total system. The most consistent complaint from all customer groups across the library was heating. Opening hours continued to be an issue to postgraduate students, particularly those using the smaller specialised branch libraries. University of New South WalesThere was considerable variation in levels of perception of service quality at UNSW between categories of users and between full and part time students. Confirming data collected through earlier focus groups, it appeared that academic staff and postgraduate research students perceived a higher quality of service than undergraduates (55.2% of academic staff and 52.1% of postgraduate research students rated the library as excellent). The students who found most to criticise in their overall course experience - full time post graduate coursework students and part-time undergraduate students were also the most subdued in their admiration of the library, with only 22.3% of full time postgraduate course work students and 23% of part time undergraduate students rating the library as excellent. Since 88.2% of the hard to please full time undergraduates, 86.5% of academic staff and 87.5% of part-time postgraduate research students gave the library a 5, 6,or 7 rating, it would appear that all key constituents see the library in a positive light, albeit tempered at the highest level. University of QueenslandThough perception of service quality levels varied across University of Queensland branches, for all branches customers were more satisfied than they were dissatisfied. In seven of the 13 branches 50% or more customers noted a perception of high service quality (6 or 7 out of 7). As with Melbourne, Queensland noted a high degree of similarity in areas of perceived lower service quality across its branches. Users were most concerned about computer workstation numbers, photocopying facilities, and prompt corrective action regarding missing books and journals. University of AucklandAgain there was a large degree of similarity in issues of concern identified across different library branches, category of library user, and faculty. Those issues included individual seating and library space adequacy, and whether prompt corrective action was taken regarding missing books and journals. Ten Highest Service ExpectationsSix of the ten items with the highest user expectations in terms of what makes an excellent library were common to user responses from all four libraries. The survey covered more than 40 items in every case. These few broadly accepted criteria for excellence were:
The Yale study also found users rated collections meeting needs, clear catalogues, equipment in good working order and expert staff important to their ideal library. The pattern for other top ten items is detailed below.
Some libraries had as many as eight of the top ten in common with another library (eg UNSW and Auckland). It is also interesting to note that usually only one or two at most of items in those top ten service expectations related to library staff and these tended not to be in the top five. Staff accuracy is the human trait seen as most important to users. The reverse position held for performance ratings where, as can be seen below in Section 7, staff items make up almost all the top ten performance items. Largest Service Expectations/Performance Gaps and Areas Identified as Priorities for ImprovementIn all four libraries priority areas for improvement strategies were primarily identified by:
There were four items with the largest service expectations/performance gaps common to all four libraries.
There were only two items with the largest service expectations/ performance gaps common to three libraries:
There were three items with the largest performance gaps common to two libraries:
The gaps were presented in slightly different ways in the four reports as the information below reveals. University of MelbourneService expectations/performance disparities to be addressed were, in order of magnitude:
The 157 comments included positive comments on services (Buddy, helpful staff and collections) and complaints about specific issues in particular libraries and/or suggestions for improvements. There were many comments on survey length, library fines, opening hours, shelving and not being able to find material on the shelves, especially in the larger libraries, photocopying, due date stamps, and lack of space in smaller libraries. Particular local concerns included small library branch closures as part of amalgamation into larger libraries. Students from these areas voiced their preference for their smaller branch library. A number of Continuous Improvement Projects were set up to deal with most of these items during 2000. University of New South WalesBased on the differences between respondents' ranking of the relative service expectations and the performance of items, a review of the following ten items occurred. It is noted that seven of these 10 priorities for improvement are in the top 10 service expectations listing for UNSW.
The 340 comments on quality, 265 compliments and 542 suggestions for improvements were principally related to library materials, then computing/ electronic issues, and physical environment. These collected comments and suggestions provided very specific data dealt with in-house. After discussion in various forums, staff work-shopped items with the largest service expectations/ performance discrepancies and came up with specific service improvement strategies. These were integrated with operational plans if not already in place. The overlap between UNSW's greatest discrepancies (priority improvement areas) and those of the other three libraries is shown below with the five in common perhaps representing the most persistent improvement challenges for academic research libraries.
Those marked # above were also identified as key areas for improvement in the Yale study. It has been a sobering experience to discover that:
University of QueenslandThe University of Queensland Library is targeting areas identified by its customers as those on which the customers feel the library can improve. These have been identified from both survey variables and the open comments section. It is to be noted that the Top 10 Improvements list contains six items from the Top 10 Service expectations list (marked *) as areas of concern to customers which are also high priority for them.
The open comments reflected survey findings. Areas of frustration included more photocopiers, more computers, longer opening hours across the system, and the need for more space in certain branches. The identified areas are being addressed. Photocopiers have been relocated for ease of access. More email terminals and 'express' workstations have been made available, and opening hours have been extended in several branches. Priority Groups have addressed the issues of missing books and journals, and re-shelving of materials. Submissions have been made for capital funding for facilities. Many compliments regarding staff service were made, confirming that library staff provide quality service is amongst the most important issues for customers and performed well by the library. University of Auckland
The Top 10 Improvements list contains three items from the Top 10 Service expectations list, the library collection is adequate for my needs, information resources (books, electronic etc) are easily accessed and photocopying facilities are adequate. These are areas of concern for customers and high priority for them. All of the eleven items least in need of improvement related directly to staff. Variables are disparate between local population groups, suggesting that the University of Auckland Library's customers hold varying views about most of the issues. The range of comments very much reflected areas where there were the largest gaps. There were many comments on seating, overcrowding, and noise. Access to collections, action on missing material and adequate material for courses also featured. There were lots of comments on the behaviour of other users eating, drinking, reserving seats, noise, and the use of cellular phones. Amongst these negatives were a large number of compliments. These also recognised that library staff were providing excellent service despite the physical restraints and shortages in resources. Comments were circulated to appropriate staff for action, and steps are being taken to make improvements. New procedures have been developed for missing material and progress is being made towards more study space by extending the existing General Library and developing a separate Information Commons. ConclusionThis exercise was notable for the focus of managerial interest directly on the 40 plus items in the survey themselves and the gaps between the users' perceived service expectations and the perceived performance of the libraries for each item. The interest in specific items was far greater than interest exhibited in the service dimensions and the range of analyses produced by dimension as part of benchmarking. This dimensional aspect of service was treated as being virtually irrelevant by the managers despite the range of analyses available as part of these reports, a continuing focus in research from Parasuraman onwards and the continuing inclusion of dimensional analysis in recent library surveys including the Yale, University of Virginia and Singapore surveys.18 In contrast to the lack of interest in the dimensions, the items themselves, their service expectations and the performance gaps excited extended debate. Users of the four major Australasian academic libraries agree to a considerable degree on what constitutes an excellent library. The users recognise that staff are central to making the most of given resources. Priority areas where users would like to see improvements usually cluster around issues perceived as important by users and also overlap to a considerable degree between Australasian academic libraries. Prior to repeating the exercise in 2001, the instrument itself will be rigorously reviewed to make the common instrument as valid and useful as possible Notes:P Hernon D Nitecki and E Altman Service Quality and Customer Satisfaction: An Assessment and Future DirectionsJournal of Academic Librarianship vol 25 no 1 January 1999 pp9-17 Universitas 21 website: http://www.universitas.edu.au Universitas 21 Libraries Group website: http://www.lib.unimelb.edu.au/u21/strat.html#Goals A Parasuraman V Zeithaml and L Berry A Conceptual Model of Service Quality and Its Implications for Future Research Journal of Marketing vol 70 no 3 Fall 1985 pp210-230 C Cook V Coleman and F Heath SERVQUAL: A Client Based Approach to Developing Performance Indicators in Proceedings of the Third Northumberland International Conference on Performance Measurement in Libraries and Information Services Northumberland 27-31 August 1999 Newcastle-upon-Tyne Information North 2000 pp211-218 S Andaleel and P Simmonds Explaining User Satisfaction with Academic Libraries: Strategic Implications College & Research Libraries vol 59 March 1998 pp156-157 P Calvert and P Hernon Surveying Service Quality within University Libraries Journal of Academic Librarianship vol 23 no 5 September 1997 pp408-415
C Cook and F Heath LibQUAL+: One Instrument in the New Measures Toolbox ARL Bimonthly Report 212 P Tan and S Foo Service Quality Assessment: A Case Study of a Singapore Statutory Board Library Singapore Journal of Library & Information Management vol 28 1999 pp1-23 C Cook and F Heath op cit
C Cook and F Heath The ARL "LibQUAL+" Pilot Project: An Update ARL Bimonthly Report 211 at
New Culture of Assessment in Academic Libraries: Measuring Service Quality Washington DC October 20-21 at
For an example of the 1999 Universitas 21 Client Perception of Service Quality Survey see D Nitecki and P Hernon Measuring Service Quality and Yale Universitys Libraries The Journal of Academic Librarianship vol 26 no 4 July 2000 pp259-273 C Cook V Coleman and F Heath op cit P Tan and S Foo op cit D Nitecki and P Hernon Measuring Service Quality... D Nitecki and P Hernon ibid
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
