AARL

From planning to improvement: Monash University Library's quality review

Marie Pernat

Abstract During 2003, Monash University Library conducted a comprehensive quality review as part of the university's program of reviews. A quality management group was established in September 2002 to drive the process. All staff were given the opportunity to contribute input to the initial self-review. Methods of collecting data, collating responses and an insight into the self-review report are provided as well as an overview of the subsequent external panel visit and report. Issues that arose, solutions to problems, consideration of the self-review process and benefits of the quality review are included. The library's approach to quality review has been recognised as best practice within the university, thus enhancing the library's profile.

In 2001, Monash University developed guidelines for review of the quality of faculty and support services and conducted an institutional self-review. The report of the self-review, Still learning: The report of our self-review [1], recommended that a schedule be published of reviews done by faculties and support services and that there be co-ordinated follow up on responses to the recommendations contained in each review report. Subsequently, Monash University Library advised the university it would undertake its quality review during 2003.

The Centre for Higher Education Quality (CHEQ) at Monash developed a quality framework [2] that, when implemented across the university, would greatly assist in preparing for audit by the Australian Universities Quality Agency (AUQA). Responsibility was devolved to individual faculties and support services to undertake a self-review and to produce a report of the findings of this internal review. Once the self-review was completed, there was a second stage to the review program that consisted of an external review panel visiting the faculty or support service (in this case the library) to conduct interviews on work unit operations. The panel then prepared its own report, informed by the self-review report, its visit to the library and interviews with stakeholders. The purpose of the panel report, with recommendations for action, was to improve the quality of library services. This approach to quality, incorporating the self-review, is also favoured by AUQA who request that a portfolio prepared by the auditee be submitted to the audit panel prior to the audit visit. [3]

A key aspect of the Monash approach to quality is fitness for, and of, purpose. Fitness for purpose means that the university must create its own quality agenda for its own unique situation and that purpose at all levels needs to be agreed. [4] Fitness of purpose applied to the library involves examination of the alignment of the library's mission with that of the university.

The library's fitness for purpose was expressed in its mission to 'enrich the learning, teaching and research programs of the university by providing seamless and timely access to high quality scholarly information and learning materials in a range of formats, in order to meet the needs of staff and students wherever they are located within the global Monash'. [5] The self-review would reflect on how the library was meeting its mission and whether the mission was still appropriate.

Staff were asked to consider and demonstrate: what they did; how and why they worked the way they did; how well, and if, processes were working and how they might be improved; and, how they themselves might improve. They readily related to the Monash planning and review cycle shown below that imparts a structured, systematic approach to bring about change.

Monash University Library comprises eight branch libraries at six campuses in Australia as well as libraries at the Malaysian and South African campuses. Over 270 library staff support the staff and students in ten faculties. The library is regarded as a leading research library in Australia and is a member of the group of eight [6] university libraries. It provides access to an extensive range of electronic resources, has developed high class print collections over its 40 year history, and its staff are recognised for their innovative initiatives. The university guidelines for review specified that the self-review was to focus on the work of Australian-based staff. There was no input from the overseas campus libraries but the delivery of services undertaken by Monash University Library for those campuses was included as part of the review.

Preliminary tasks

In September 2002, the library's quality management group (QMG) was formed. Members were the university librarian, the five library directors (one of whom was the project sponsor), a CHEQ representative, and the project manager, who was a senior librarian already employed in the office of the university librarian. Thus the project had high-level input and support and the project manager would be able to spend the necessary time on the project.

The purpose of the QMG was to:

• integrate and ensure quality assurance improvement is embedded into Monash University Library activities,
• use the university's quality review process to improve the effectiveness of the library,
• work within the Monash quality cycle guidelines - plan, act, evaluate and improve,
• adopt a communication strategy on quality, and
• adopt a program of service improvement.

At an early stage, a project plan was prepared that identified the timelines, tasks of the project manager and involvement of staff, possible risks, training and funding requirements. A communication strategy was devised to ensure all library staff, panel members, interviewees and university staff would be aware of progress and outcomes. This included regular e-mails from the university librarian, directors' reports to staff within divisions, and the introduction of a library web page [7] on quality activities. The university librarian invited all library staff to contribute to the self-review.

Just prior to the first meeting of the QMG, a member of CHEQ addressed the library management committee to explain the university's approach to quality and the rationale of the review process. The QMG met formally on six occasions during the review period. Informal reports of progress were made at staff meetings and advice was sought from CHEQ between meetings.

A major task that required considerable effort by the QMG was to finalise the terms of reference of the review. The library operates with a matrix structure whereby for some staff, faculty responsibilities can overlap with branch based responsibilities. It was decided that the eight branch libraries and the centralised divisions and units would be asked to provide input.

The terms of reference (headings of the report) were aligned with the university's model. Some minor changes were made to reflect library practices in more detail, for example, cross campus faculty team operations, information literacy classes, and collection management. The key headings were grouped as follows:

• organisational structure, management, quality assurance and improvement,
• human resources,
• core services - library resources; library services; physical infrastructure (including IT), and
• professional and community activities.

Details of the scope of these terms of reference, documentation and the types of evidence to support the findings for the self-review report are found in the Quality review 2003 summary document. [8]

Project milestones

Self-review process

The documentation prepared by CHEQ on each stage of the review process provided a framework within which to act, commencing with the values and principles referred to previously. The library also used a set of questions, provided by CHEQ, to assist staff to identify activities related to the terms of reference, relevant documentation and evidence. The questions were modified slightly to include terminology and functions appropriate to a library.

The project manager prepared a PowerPoint presentation for senior library staff that explained the university's quality values and principles, and the library's approach to self-review. This generic PowerPoint was significant because it ensured all managers were provided with a consistent approach and would have a similar understanding of the processes to communicate to their staff. The presentation included key messages that the university librarian wished to convey to staff about to undertake the review. The background to the review process at Monash, the library's time line and the role of AUQA were explained. The objectives of the library review were to:

• make a genuine attempt to identify strengths and weaknesses and areas for further review,
• focus at a high level, with a concentration on analysis, not solutions, and
• produce an objective and open report.

Ground rules were established. The roles of individual staff were not to be examined. The focus was to be on the quality of work processes. Although all the ideas of staff would be considered, this did not guarantee that they would be incorporated in the final report. The emphasis was to be on analysis and evidence available and not on finding solutions at this time. The self-review report would be a public document, to be made available on the library web site.

Staff responded willingly to this approach and formed small work groups to consider how well their units were achieving their roles and fitness for purpose. Different methods were used to provide input onto the template designed by the project manager, which mirrored the terms of reference for the overall report. In some units, staff jointly contributed. In other cases, the director provided a first draft and staff then contributed. Within larger divisions, small workgroups gave separate reports that were consolidated into a divisional response. Individual staff input was also received.

The library did not specifically seek the views of students and staff for input to the self-review. However, a number of instances of pre-existing library surveys and examples of customer feedback were referred to in the self-review report to demonstrate that the library does actively seek such input as part of its regular activities and uses it to inform decision-making and planning.

Self-review report

Staff from 18 library work areas provided 163 recommendations for further action to improve the library. A number of these issues had not previously been raised and were included in the report for consideration by the external panel. Other task-based, operational matters were resolved quickly and directly with supervisors.

The report, drafted by the project manager and finalised by the QMG, consisted of 11 800 words, plus appendices. Included were key issues, strengths and 28 opportunities for improvement relating to the terms of reference. The appendices included ten attached documents, a list of additional materials that would be made available to the panel, and the abbreviations used in the self-review report. [9] All sources referred to in the report were fully referenced. The gathering of documents and evidence for the self-review report and for the panel visit was a time-consuming task that required advance planning.

Benefits of the self-review

The self-review process provided the opportunity to embark on an exercise to effect improvement that had not been previously attempted by the library. Staff who participated gained familiarity with the cyclical nature of the approach to quality at Monash and applied that approach when analysing the work of their units. Newly appointed directors and staff gained a timely insight into the library overall. The self-review report reflected the input of many staff and therefore staff felt ownership of the opportunities they identified to improve the quality of services to library users. The self-review report served to inform the panel in detail of the library's view of their operations prior to the panel's forthcoming visit as part of the review process.

External panel review process

The project manager also acted as the contact person and provided secretarial support for the external panel. One essential task was to ensure that panel members were selected about six months prior to the panel's visit to the library. CHEQ provided guidelines on the composition of the review panel. The final decision rested with the university librarian, who then sought endorsement from the DVC (academic). The panel for the library review comprised: a Monash dean (chair) experienced in international operations; a Monash professor well-versed in both research and teaching; a Monash professor with expertise in graduate studies and flexible delivery of courses; the executive director of Monash information technology services; and two university librarians experienced in quality reviews from the group of eight universities. Panel members were provided with advice on the review process at Monash, particularly relating to the panel visit and report that would be produced.

The panel requested some additional information, including more recent statistics on the library's staffing and acquisitions budgets and those same figures for the group of eight university libraries. Also requested were the customer survey results that had become available after the self-review report was written and charts that showed Monash University Library's performance in the survey compared to other group of eight libraries. This information was not held by the library and had to be obtained at short notice.

External panel visit

Three weeks prior to the panel visit, the chair and panel members were asked by the library to finalise the list of persons they wished to interview so that appointments could be organised. At the first panel meeting on site, expected outcomes of the panel review were clarified. Agreement was reached on methods of questioning to be used during the interviews scheduled and on the particular aspects assigned to panel members, given their areas of expertise. The format and headings of the panel report were agreed.

The panel visit took place at Clayton campus, commencing late afternoon on the first day and then continuing for a further two full days. The panel conducted ten interviews with 23 stakeholders - deans, senior administrators, academic staff, library staff and students. In order to obtain a collective view of student opinion, student representatives were interviewed. All interviewees had been briefed on the process by the project manager and provided in advance with a copy of the self-review report. It was important not to crowd the itinerary because the panel needed flexibility to change the program, to review previous sessions and consolidate the report's findings progressively. Small group interviews of three or four persons worked well. Sections of the Matheson Library (humanities and social sciences) and Law Library were visited. The panel did not wish to tour facilities or attend presentations unless there was a particular issue they needed to probe or they required an explanation of how potential problems they identified might be being addressed.

Examples of panel requests during the visit included:

• loans statistics over several years (to determine trends in the borrowing of print materials),
• the number of volumes housed in one of the library buildings (concerning space issues),
• the terms of reference of library user committees (relevant to the extent of communication between faculties and the library),
• a checklist of offshore services (delving into service agreements with overseas partners), and
• postgraduate focus group input (seeking further information on students' opinions on online services).

External panel report

The panel's report consisted of 8000 words, plus four appendices. The appendices comprised the previously mentioned tables of data on group of eight funding statistics and comparative customer survey ratings that were additional to the self-review report; a list of all documentation provided to the panel; and the panel membership. The final draft of the report was provided to the university librarian to check for factual accuracy.

The report contained six commendations. These affirmed the university librarian's recently implemented strategic planning process and the revised library management structure. Support was also expressed for the innovative approaches to the delivery of electronic resources and services including the electronic readings and reserve lists. There were 18 recommendations, which, in the view of the panel, if put into practice would raise the quality of library services.

The panel recommended some additional benchmarking and suggested instances of joint activity with other university libraries. Although a high-level, helicopter view was taken of library operations, the panel did home in on several areas including: the lack of access to senior policy-making forums available to the university librarian; the effectiveness of the 'one library' system in a multi-campus university that offers diverse courses to students with diverse needs; the standard of the libraries' physical facilities; the need for further staff training in order to take best advantage of the electronic environment in which they work; and the extent of library involvement in overseas partner arrangements.

The recommendations, in essence, confirmed many of the opportunities for improvement that had been identified for action by staff in the self-review report. The panel did suggest some new strategies and decided that some activities should be addressed more urgently than the library had stated.

Benefits of the external panel review

The panel's report gave an external perspective on library operations that had not previously been provided. The opinions of senior university staff and students that were expressed to the panel during the visit confirmed the feedback the library had been receiving from its stakeholders through other channels. The recommendations for improvement were well received by library staff and action was quickly taken in those areas identified as high priority. As the report also highlighted best practice library operations and acknowledged areas of staff expertise, it was a positive report for staff overall.

Post review actions

In order to document follow-up actions on the panel's recommendations, and to assist with meeting university reporting requirements, a post review action plan was drawn up using the headings shown below, under which the panel's recommendations were individually listed.

Some recommendations had implications beyond the library, for example, recommendations on additional funding for the library, and on a senior university forum for attendance by the university librarian. There was no additional budget available for implementing review recommendations but cases could be made and funding sought to progress projects of benefit to the university. Some recommendations named other support services and could have wider resource implications. CHEQ developed guidelines for handling such recommendations.

In an effort to monitor the activities of units that undertook self-reviews, the university included the requirement of a report to the vice-chancellor's group (quality) six months after review on the progress made in implementing the recommendations. The university librarian was invited to meet with the group to provide an overview of the library's review processes, including benefits and problems. The university librarian was also asked to report on actions taken on panel recommendations that were considered to be of possible immediate impact or of strategic importance to the university.

Examples of actions progressed following the panel's recommendations included:

• a review of library user committees (with input sought from faculties),
• a submission for, and receipt of, additional funding for library collections in 2004 (also identified as a priority before and during the self-review),
• the inclusion of the university librarian as a member of the university's senior management committee (a new initiative implemented by the vice-chancellor),
• an audit of staff and service levels across all library branches,
• an audit of staff training needs and a subsequent training plan,
• further collaboration with the university's information technology services in the provision of help services, and
• implementation of the library's facilities master plan and contribution of funds from the university to commence the refurbishment of libraries.

Enduring outcomes

A major challenge upon completion of such a review program is to embed quality systems to track progress, and to ensure the processes introduced are worthwhile. In this regard, a number of actions have been taken to instil a culture of analysis and review. They include:

• The role of quality co-ordinator is now ongoing, with the brief to co-ordinate progress on the review recommendations; ensure inclusion of some initiatives in planning documents; track AUQA good practice and audit reports and advise on Monash's preparation for audit; and monitor library statistics and update web materials.
• The QMG continues to meet to improve quality by reviewing key performance indicators in service level agreements, analysing survey results, and monitoring the recommendations of the panel.
• Quality-related clauses are specified in strategic initiatives in divisional, unit and personal performance plans.
• Quality-related issues are included as standing agenda items at committee meetings and there is regular reporting of outcomes.
• Actions identified for implementation are incorporated in annual budget and project planning cycles as appropriate.
• The library is represented on university quality groups and therefore has a forum to contribute to the wider university, to share experiences, and to be informed on quality matters as they arise across the university.

That staff now have improved understanding of quality in the library was demonstrated in the staff survey held in September 2003, when, compared with the previous survey in 2001, there was a significant improvement in the performance of 'continuously improving the way we do things'. This finding was further tested in May 2004 when staff were asked for evidence of what has changed as a result of the quality review. From the many responses received, several shown below illustrate improvements at each stage of the quality cycle - plan, act, evaluate and improve.

Staff are now continuously planning and reviewing their work and allocating time to do so.

Additional door statistics are being collected to get a better idea of usage patterns and appropriateness of opening hours, in order to review them.

There is commitment to the process of analysing work areas and producing quality documentation, and more statistical data is collected. Staff have better understanding of why they should do this, and can see the results of better planning.

The quality review process reinforced continuous improvement and staff focus on the need to review practices to ensure high standard services to our clients.

Issues that arose and how they were addressed

• The perception by some staff that the self-review would impact considerably on day-to-day library operations.
  The project manager organised the project and had responsibility for most tasks. Library staff efforts largely consisted of an initial meeting to understand the process, followed by one or two meetings to prepare content, and finally, written input on the self-review template. All units were requested not to write more than three pages, and to list, not supply, relevant documents. There was a continuing need for a project manager throughout the review period who at some stages needed to attend to review-related matters full time.
• The belief that some issues previously identified would remain unresolved.
  Staff were advised that the self-review could not solve all their problems. However, with their contributions, issues would be raised and the panel would also then have the opportunity to highlight some of them to the university librarian. Staff were encouraged to document their concerns so that, through the gathering of input from all units, any repeated and widespread problem areas would be identified.
• Scepticism as to whether the self-review report would include all issues raised.
  Staff were advised that every effort would be made to present a balanced picture of library operations, as reported by them. However, the review was to be at a high level and operational issues would not be included. They could be discussed directly with supervisors.
• Requests by the panel for additional documentation and factual input at the time of the panel visit.
  The panel secretary responded to a number of requests for data and information from the panel. It was important that administrative staff were on hand to make changes to appointments, photocopy and perform other ad hoc tasks that arose during the course of the panel visit.
• The need for the panel to ensure interviewees did not feel their personal opinions were being closely examined.
  The panel's guidelines specified suitable questioning techniques. Whenever possible, at least two persons were interviewed together. The purpose behind the questions asked was explained to interviewees. Their names were not recorded in the panel report.
• The limited time available during the panel's visit to consider library operations or to visit all campuses and meet the staff.
  The panel identified the areas on which they would focus and asked similar questions of interviewees from different campuses. Some panel members were familiar with campuses other than Clayton.

Reflecting on the review process

The congruent nature of the reports emanating from the library's quality review - the library's own 'examination' of its activities and the external panel's subsequent findings - might suggest the two-stage review process has limited effectiveness, and perhaps also some constraint on the part of the panel to advise on new directions for improvement. However, the author believes that the expert panel's considered perspective added the stamp of authorisation. The panel did seek further input from students, staff, administrators and library staff in relation to the library's fitness for purpose documented by staff in the self-review report. The end result was a series of recommendations and suggestions to improve library services, many of them previously identified, and therefore owned, by library staff. Without such a thorough, honest self-review document to inform them, the panel would still have had the opportunity to delve into library operations and 'test' the library's premises as they saw fit. There might have been fewer commendations and recommendations as a consequence because the length of the panel's visit was very limited. The self-review document, prepared from library staff input and supported by evidence of how well or otherwise processes were being undertaken in the work unit, is a more effective way to demonstrate the quality of library operations to a panel than a document with descriptions of services and lists of statistics.

When undertaking future self-reviews, it would be appropriate to seek the views of students and staff at the time of review in order to have very recent opinion available to further substantiate claims made in the self-review document. Feedback could be sought, written submissions requested, possibly via the web and e-mail, from groups and individual stakeholders, specifically on how the library is performing in relation to the review's terms of reference. Student and staff focus groups and forums could be organised.

It is not suggested that the quality review process would be the most appropriate way to resolve difficult operational issues or that it should be used for recommending major changes to staff structures. Further, if some sections of a work unit do not contribute to the self-review report, the panel's task will be difficult, given that they are external auditors whose task is largely based on the interpretation of the evidence provided to them by that work unit. The panel's findings might lack credibility with some staff and opportunities identified by the panel to bring about improvement might be difficult to implement.

Overall Benefits of the Library's Quality Review Program

• Areas for improvement identified by staff and confirmed by the external panel,
• commitment to ongoing improvement,
• increased awareness of the approach to quality at Monash University,
• gaining of a readily-understood model that places quality within a practical framework,
• a planned and systematic approach,
• analysis of library services undertaken,
• staff involvement and ownership of the report,
• recognition of library good practice by the university,
• an opportunity to share experiences and collaborate on quality-related programs with other support services,
• increased stakeholder satisfaction in the long term as a result of changes implemented, and
• appropriate preparation and readiness for AUQA audit.

Conclusion

The leadership role of senior library staff and the decision to appoint a project manager to oversee and co-ordinate the review at all stages were significant factors in motivating staff to complete the task. The successful outcome was due to the comprehensive input from library staff themselves and the substantial and useful findings of the external panel. As a result of the thorough internal review and subsequent external panel review of their work practices, library staff now have a greater appreciation of quality systems and a clear framework within which to pursue improvement.

The library intends to conduct a similar quality review every five years. In the meantime, stakeholder feedback will continue to be regularly sought, staff trained in new technologies, and changes introduced for the benefit of library users.

Notes

1. Monash University Still Learning: The Report of our Self-Review http://www.adm.monash.edu.au/unisec/res/Still%20Learning.pdf [accessed 8 November 2004]
2. Centre for Higher Education Quality Quality at Monash: Values and Principles http://www.adm.monash.edu.au/cheq/documents/Quality%20at%20Monash%20-%20Values%20and%20Principles.pdf [accessed 8 November 2004]
3. Australian Universities Quality Agency Audit Manual v 2 (April 2004) http://www.auqa.edu.au/qualityaudit/auditmanuals/auditmanual_v2/AUQA_Audit_Manual_v2.pdf [accessed 8 November 2004]
4. Centre For Higher Education Quality op cit
5. Monash University Library Strategic Plan 2004-2006 http://www.lib.monash.edu.au/plans/2004/ [accessed 8 November 2004]
6. The Group of Eight Web Page is at http://www.go8.edu.au/ [accessed 8 November 2004] 7 Monash University Library's Quality Web Page is at http://www.lib.monash.edu.au/quality/ [accessed 8 November 2004]
8. Monash University Library Quality Review 2003 Summary Document http://www.lib.monash.edu.au/quality/QualityReview2003.html [accessed 8 November 2004]
9. Monash University Library Quality Self-review Report http://www.lib.monash.edu.au/quality/self-review-report.pdf [accessed 8 November 2004]