The Australian Library Journal

Evaluation of the impact of the 2003 Aurora Leadership Institute - 'the gift that keeps on giving'

Kay Barney

Manuscript received December 2003

This is a refereed article

Leadership, which barely rated a mention in the library and information management literature twenty years ago, has become a 'hot topic' over the past decade. The current literature identifies serious concern about leadership in the library and information profession (Corrall 2002; Riggs 2001). In Australia and New Zealand, a generation of library leaders from the 'baby boomer' generation is nearing retirement, a trend reflected in many other countries including the United States of America and the United Kingdom. The concern is: who will fill these positions? It will need to be people with the leadership skills to deal with constant change and the complexity of the library and information profession. Challenges include dealing with the doubling of scientific knowledge every ten years, of computing power every eighteen months and of the internet every year (Paterson 1999, p144).

The need for more people to undertake leadership training is now widely acknowledged in the profession. Corrall (2002) notes that some library managers - herself included - have undertaken a Master of Business Administration (MBA) program to gain a wider perspective on leadership issues. However, such courses often cost more than $20 000, which would preclude many with leadership potential from benefiting from this option. Prior to 1940, the view in the literature was that leaders were born and not made. However, research has since shown that leadership skills and attitudes can certainly be acquired (Sogunro 1997; Stogdill 1974). Schreiber and Shannon (2001) quote leadership guru Warren Bennis: '[Leadership]... is something that can be learned by everyone, taught to everyone, denied to no one.' During the past ten to fifteen years, new programs have been developed around the world in response to the growing demand for leadership training. There are more than twenty in the United States.

In Australia, just one program offers specific leadership training for the library and information profession - the Aurora Leadership Institute - which began in 1995 'to assist future leaders to recognise and develop their leadership abilities'.

The Aurora Leadership Institute

The Aurora Leadership Institute is an annual six-day residential leadership training program for people in all types of libraries in Australia and New Zealand. It targets those who are generally five to ten years into their library careers. Applicants are nominated by their organisation and must provide a letter outlining why they want to attend. Six Aurora Leadership Institutes have been held in Australia, usually at Thredbo, New South Wales. The 2003 Institute was held in Canberra because bushfires closed Thredbo. The Institutes are modelled on the US-based Snowbird Leadership Institute, held at Snowbird near Salt Lake City (Neely & Winston 1999, p413). Aurora uses the same facilitators as Snowbird to conduct the Institute - John Shannon and Becky Schreiber of Shannon and Schreiber Associates - employing experiential learning techniques in four groups comprising eight participants (thirty-two in total) and two mentors in each group.

The Aurora Foundation Ltd, the organising body, is an incorporated not-for-profit company. Aurora Foundation Ltd Board members, the late Warren Horton (then chairman) and Ian McCallum (secretary), believed that the value of the Aurora Leadership Institute lay in the knowledge and skills it imparts to both participants and mentors to recognise and develop their leadership abilities, and in the professional relationships it fosters. Mentors are a key part of the Institutes, and were selected and invited personally by Warren Horton, the former director-general of the National Library of Australia. The mentors act as facilitators, role models and guides, working closely with participants throughout each Institute. They also each present a session called 'My practice, my passion' in which they talk about an episode in their professional lives that gives insight into their professional commitment.

Defining leadership

While leadership is widely acknowledged as a desirable skill, there is often confusion about exactly what leadership is. Riggs (2001, p5) says there are at least 100 definitions of leadership. Warren Bennis (1989, p7) makes the following points about the difference between managers and leaders:

The manager administers; the leader motivates.

The manager is a copy; the leader is an original.

The manager focuses on systems and structure; the leader focuses on people.

The manager relies on control; the leader inspires trust.

The manager has his eye on the bottom line; the leader has his eye on the horizon.

However, others argue that increasingly the distinction between managers and leaders is becoming irrelevant in the context of downsizing and flatter organisational structures (Corrall 2002; Rochester & Nicholson 1998). This implies that organisations will have to realise that every employee is a potential leader whose leadership ability should be developed (Sogunro, 1998). Sogunro's study concluded that leadership can be made more effective if organisations are concerned with the training of all their members, rather than training just the few designated leaders. This has implications for the selection of people to attend leadership programs - you do not necessarily have to secure the 'best' people to attend in order to achieve a positive result. Research by Khurana (2002) shows no conclusive evidence that charismatic leadership affects an organisation's performance. Leadership, he says, cannot be isolated from the organisation and cultural environments in which people operate. This would seem to suggest the need for more leadership training so that organisations can promote from within.

Evaluating the impact of library and information management leadership programs

There is little research on the impact of library leadership training programs, even though there are more than twenty such programs in the USA. Furthermore, the research that has been conducted to date is inconclusive about whether library leadership programs are successful in developing leaders (Neely & Winston 1999; Nichols 2002; Weaver & Burger 1991). Even looking outside the library profession, the situation is not very different. The community development sector in the United States and Canada is an area where leadership training programs are booming with an estimated 650 to 750 community leadership programs operating in the United States. Some research has been done to evaluate their impact, yet even these programs are under-researched (Fredricks, 2001). Questionnaires administered at the end of the programs provide little information about the effect of the program on participants' behaviour on-the-job.

The lack of research may be due to the difficulty in measuring the impact of programs. While tests can be used to evaluate training programs that teach hard skills, leadership qualities and characteristics, often characterised as 'soft skills', are generally evaluated through anecdotal observations and/or perceptions (Sirianni & Frey, 2001). This involves asking leadership program attendees whether they think their skills/knowledge/behaviour have improved in certain areas, and using the perceived change as the criterion for determining the impact of the program. This approach is given validity by the concentration of leadership training programs on developing participants' self-perception of leadership behaviour, not increasing their basic knowledge (Brungardt & Seibel 1995).

When evaluating leadership programs, it is easiest to establish evidence of the level of participant's satisfaction with the program and most challenging to determine the level of impact that participation in the program had on the organisations in which they hold leadership roles (McLean & Moss 2003).

The challenge in assessing the impact of leadership programs is compounded by the difficulty of attributing any changes to the program (McLean & Moss 2003; Paterson 1999). Acknowledging the difficulty, or even impossibility, involved in obtaining absolute proof of the impact of a program, we have to be satisfied with supporting evidence instead (Kirkpatrick 1994, p68).

Evaluation of the Aurora Leadership Institute

The Aurora Leadership Institute can already claim success in that library and information organisations in Australia and New Zealand continue to sponsor people to attend the Institutes, at not inconsiderable cost, and demand exceeds the number of places available. There were forty-eight applications for thirty-two places for A6: the cost to the sponsoring organisation was $3410 for each candidate plus any air travel and time away from work. (For the 2004 Institute, the cost is $3800). Yet there are no objective data on the Institute's impact and value, although the literature does provide descriptive information about how Aurora was conceived (Horton, 1996) and how some individuals have responded to the experience (Dan 1996; Gow 1996; Lilley 2003; Sutherland 2003).

Although participants complete feedback forms at the end of each Institute to evaluate the course itself (as well as rating each day on a scale of 1 to 5), a more pertinent question to be answered is how have participants been able to take what they learned at the 2003 Aurora Leadership Institute and apply this to their professional lives to provide leadership in the profession? Hence this research, the primary focus of which was to analyse the short-term impact of the learnings from A6 (held in February 2003) on the participants, and their perceptions its value to their work and careers. The study includes mentors as well as participants, because many of the mentors commented at the end of the 2003 Aurora Leadership Institute that they felt they had learned more than the participants. Warren Horton attended A6, but was not surveyed because he fulfilled a different role - that of 'meta-mentor' who acts as an advisor to the other mentors, as well as to the participants.

Method

The survey methodology was used and data collected by e-mailing the questionnaire to the thirty-two participants and eight mentors who attended the Aurora Leadership Institute held in February 2003. Respondents had the option of replying by e-mail, or if they preferred to answer anonymously, by mail. No one chose to reply anonymously. This reflected the lack of controversial questions and the provision for voluntary responses. Many of the comments provided were personal reflections, indicating a willingness on the part of respondents to respond thoughtfully and genuinely to the questionnaire. It elicited a strong response rate - twenty-seven out of the thirty-two participants and six of the eight mentors. Respondents completed all multiple-choice questions and the majority commented on each of the six open-ended questions. Of the five participants who did not respond to the questionnaire, one was on maternity leave and two could not be contacted as they had moved jobs since attending A6. Analyses were conducted using SPSS 11.0.

The survey instrument was designed to look at the demographic backgrounds of participants, their years of professional experience and their perception of the short-term impact of A6 on their work and careers. It included a number of open-ended questions to elicit deeper insights into areas of particular interest to the Aurora Foundation Board, such as how A6 had affected their subsequent leadership behaviour and career aspirations. The questionnaire was partly based on that used in the US Snowbird study by Neely and Winston (1999): basic demographic queries on age, gender, type of library and years of professional library experience and items on participants' perception of the impact of A6 on their careers and professional development, were taken directly from this instrument or adapted. The questionnaire for this study did not include the items about career progression used in the Neely and Winston survey because the Aurora study looks only at the participants from the 2003 Institute, and therefore it is too early to assess the effect on people's careers from attending A6.

The questionnaire for this study was pilot tested on a group of University of Canberra postgraduate students and was also reviewed by Warren Horton and Ian McCallum. Individuals were asked about the value of A6 to their work, and to their career, using a Likert-type scale. Two items asked individuals about the value of A6 to their work and to their career. This was done using a four point scale ranging from 1 (great value) to 4 (no value). Two items asked about the extent to which their interactions with other participants and the Aurora mentors contributed to the quality of their experience. These items used a three-point scale ranging from 1 (to a great extent) to 3 (not at all).

Although the survey relies on respondents' self-reports, the questionnaire also included six open-ended items that asked respondents to provide examples of the impact that A6 had on them, their level of professional activity and their career. It was hoped that this would enhance the validity of the data by causing respondents to think more carefully about their responses by requiring them to express their thoughts in written form. Five items asked individuals to indicate their gender, age group, years of experience as a professional librarian, type of library in which they worked when they attended A6, and whether they were a participant or a mentor at A6.

Selected findings

A6 demographics

Aurora recruits participants from all types of libraries. Of the twenty-seven respondents, there was a predominance of librarians from public libraries (twelve), and from academic libraries (nine). There were three from special government libraries and three from state/national libraries. There were no librarians from schools or from special libraries in the private sector at A6.

Although Aurora usually targets participants who are five to ten years into their professional library careers, this is not a firm requirement for selection, as can be seen from Figure 1. There were nine respondents to the survey who had eleven or more years experience as professional librarians.

[mean 8.6, standard deviation 4.95]
Figure 1. Participants' years of professional library experience

The age of participants indicated roughly even groupings in each of the age groups under 45 years of age (see Figure 2).

Figure 2. Participants' age groups

There were six male and twenty-six female participants. All the males and twenty-one females responded to the questionnaire.

Perceived value of A6

Since the purpose of the Institute is to develop leaders for the future, it is critical that the program has lasting effects. The responses show that six months after A6, the majority of participants perceive a strong legacy from the Institute, which continues to influence their decisions about work and career. The perception of positive benefits flowing from A6 is important because participants who have a positive reaction to the training are more likely to learn and to transfer the learning to their jobs (Kirkpatrick 1994). A6 was perceived by 100 per cent of respondents as beneficial to their work and careers (Tables 1 and 2). The majority of respondents expressed high levels of satisfaction in terms of its impact on their work and careers. A large majority also valued the interaction with other Aurora participants and with the mentors.

In contrast with the participants, who found greater value in A6 to their careers than to their work, five of the six of the mentor respondents rated it of great value to their work but only two of the six rated it 'of great value' to their career. This would seem to reflect that the mentors are at the peak of their careers, but can still benefit in their working lives from refining their leadership skills.

Table 1. How participants rated the value of A6 to their work

Table 2. How participants rated the value of A6 to their career

Although the population of thirty-two participants and eight mentors who attended A6 is too small to produce cross-tabulations with any statistical significance, the general trends indicated by the quantitative analysis were backed up by the written answers to the open-ended questions in the survey. The analysis showed a trend for participants with less than eleven years experience in librarianship to perceive greater value in A6 to their work and careers, compared with their more experienced counterparts (see Tables 3 and 4). This is not unexpected, since those with more experience probably had already acquired leadership skills and knowledge on-the-job and attended previous courses that covered some aspects of leadership training. One of the participants said that continuous training over a period of years, including a senior management course, meant that 'the impact of Aurora was less dramatic'.

This does raise the question about whether Aurora should restrict its recruitment to early-to-mid career people, like the Snowbird Institute which targets those who are at a relatively early point in their library careers (Summers & Summers 1991, p38). This would also depend on whether the Aurora Foundation Board feels this finding for A6 would apply more broadly to other Aurora Leadership cohorts.

Table 3. Participants' experience and the value of A6 to their work

Table 4. Participants' experience and the value of A6 to their careers

The analysis also found some evidence that the librarians from public, state and national libraries found more value in A6 to their careers than did the academic and special government librarians (Table 5). This could be because public librarians have until quite recently, due to lack of funding, been under-represented at Aurora Leadership Institutes compared with their academic counterparts. Therefore, they may value the opportunity more and work harder to get the maximum benefit from the Institutes.

Table 5. Type of library participants worked in at time of A6 and their perceived value of A6 to their career

A strong link was found between attendance at A6 and a subsequent increase in their level of professional activity, indicating that A6 had already had an impact (Table 6). An increase in activity might be reflected in the number of committees they joined, or an increase in their participation on existing committees, or some combination of the two. This study focused however on the change in the level of activity, rather than attempting to compare levels of activity between participants, which might merely reflect that those who had been in the profession longer generally had higher levels of professional activity. Mentors generally reported the same high level of activity before and after attending A6. This was expected because the mentors are leaders in the profession who are already aware of the benefits and responsibilities of professional association membership and activity.

Table 6. Changes in professional activity after A6 - participants and mentors

Mentoring was shown to be a particular strength of A6. The Aurora Foundation Board considers the mentoring component to be a key part of each Institute. Nearly ninety per cent of respondents said the interaction with mentors contributed to the quality of A6 'to a great extent' and the remainder said 'to a moderate extent'. This finding reflects the emphasis that Aurora places on the mentoring component.

Discussion

In a future study, it would be useful to ask participants whether they had maintained contact with their A6 mentors, and with other participants, to see whether professional networks had been established and sustained as a result of attendance at the Institute. Although it is too soon to tell the effect of A6 on participants' careers, four of the participants mentioned that they had moved into higher positions since attending Aurora and another three said they were actively applying for other jobs as a result of attending. They cited their participation as influential in their decisions. When assessing the impact of A6, a question arises about whether a sufficient amount of time has elapsed for behaviour and attitudinal changes to demonstrate increased leadership capacity in the participants: it would be useful to study all the Aurora alumni since 1995 to see whether they have become leaders in the profession.

There is also a question about whether the activities and benefits resulting from attendance at A6 can necessarily be attributed wholly or in part to the Institute, given that the participants were chosen to attend on the basis of their leadership potential and may well have taken on leadership activities and career progression anyway. While it is not possible to separate the effect of A6 from everything else that has occurred in participants' lives over the past six months, participants provided compelling evidence that it enhanced their leadership abilities, knowledge and attitudes. Comments included:

'My perceived career path has changed trajectory, becoming higher both long- and short-term. I am keen to take on more responsibility earlier, and to make a more significant contribution throughout my working life.'

'I am working at a different level - taking more of a place amongst the 'big guns' in terms of policy and strategic alliances.'

'I'm more self-aware and determined to look at the future and bigger picture - where are we going - rather than just short-term solutions to any issues/problems.'

'[I am] more confident in my leadership ability. I observe people more and pick up on their interests to drive an innovation or project.'

This study is not able to demonstrate conclusively that A6 has resulted in behaviour changes and any consequent impacts on organisations and communities, but this is not surprising due to the difficulty inherent in measuring such change, as found by McLean and Moss (2003). However, it does provide many examples of leadership activity undertaken by respondents that they attribute (directly or indirectly) to their attendance at A6, reflecting changes in their behaviour, attitudes and knowledge. Participants variously reported how they had gained confidence in their abilities and ideas, put into practice new leadership skills, applied for promotions, have increased enthusiasm for, and increased commitment to, the profession.

While this study used a self-evaluation approach, a possible alternative for future research would be to ask the nominators of the participants to comment on their performance at work post-Aurora. This could provide useful indicators that were not possible within the scope of this study.

It is worthwhile noting that the Aurora Leadership Institutes have had the same facilitators since inception - John Shannon and Becky Schreiber. They provide continuity and consistency in terms of course content, so that every participant who attends an Aurora Leadership Institute has the same experience which promotes cohesion among the group. In addition, the mentor selection policy has been to choose some mentors who have previously mentored at Aurora, together with some who have not mentored before, thereby increasing the pool of available mentors over time. This too, has provided continuity and consistency for the program. If there is a change of facilitators in the future, this could impact on the program itself and how it is perceived.

Conclusion

Warren Horton and Ian McCallum believed that the findings of this study will further substantiate Aurora's high reputation in the community and encourage even more people to nominate for the program, thereby raising standards in the profession even higher. The findings of imply that demand for places will increase over time as the benefits derived from attending an Aurora Leadership Institute are even more widely recognised. Like A6, the 2004 Institute at Thredbo, New South Wales, in February 2004 is oversubscribed, with fifty-six applicants applying for the thirty-two places. This suggests an unfulfilled need for leadership training in Australia and New Zealand that would justify running the Institute more frequently or increasing the number of places available at each which in the past has proved unwieldy. As the number of Aurora graduates increases, a further effect will be to raise awareness of the importance of leadership in the profession in Australia and New Zealand. This will raise the standard for everyone, whether or not they have attended an Aurora Leadership Institute.

Summing up, one participant called it 'the gift that keeps on giving' and another wrote:

At the end of Aurora I felt that I had been given a gift - support, ideas, a fresh perspective, time to focus on and embrace my strengths and lots of ideas and affirmation. I believe I am better at my job as a result.

References

Bennis, WG (1989) 'Managing the dream: leadership in the 21st century' Journal of Organizational Change Management vol 2 nº 1, pp6-10.

Brungardt, CL and Seibel, N (1995) 'Assessing the effectiveness of community leadership programs' Kansas Leadership Forum Publication Series, Kansas Rural Development Council, Topeka, KS.

Corrall, S (2002) Developing library leaders: a management responsibility, viewed 20 March 2003, http://cilip.org.uk/about/president_1016.rtf

Dan, K (1996) 'One perspective' Australian Library Journal vol 45 nº 1, pp21-22.

Fredricks, SM (1998) 'Exposing and exploring state-wide community leadership training programs', Journal of Leadership Studies, viewed 25 April 2003 Expanded Academic ASAP database.

Horton, WM (1996) 'It all began over dinner' Australian Library Journal vol 45 nº 1, pp12-16.

Gow, E (1996) 'Aurora Leadership Institute: an assessment' Australian Library Journal vol 45 nº 1, pp19-20.

Khurana, R (2002) 'The curse of the superstar CEO' Harvard Business Review vol 80 nº 9, pp39-45.

Kirkpatrick, DL (1994) Evaluating training programs: the four levels. San Francisco: Berrett-Koehler.

Lilley, S (2003) 'Aurora Recall: some reflections' Australian Library Journal vol 52 nº 2, pp118-119.

McLean, S and Moss, G (2003) 'They're happy, but did they make a difference? Applying Kirkpatrick's framework to the evaluation of a national leadership program', The Canadian Journal of Program Evaluation vol 18 nº 1, pp1-23.

Neely, TY and Winston, MD (1999) 'Snowbird Leadership Institute: leadership development in the profession' College and Research Libraries vol 60 nº 5, pp412-425. Also presented as a paper at the Association of College and Research Libraries 9th National Conference, 8-11 April 1999, Detroit, Michigan.

Nichols, CA (2002) 'Leaders: born or bred? Confessions from a leadership training junkie' Library Journal vol 127 nº 13, pp38-40.

Paterson, A (1999) 'Ahead of the game: developing academic library staff for the twenty-first century' Librarian Career Development vol 7 nº 12, pp143-149.

Riggs, DE (2002) 'The crisis and opportunities in library leadership' Journal of Library Administration vol 32 nº 3/4, pp5-19.

Rochester M and Nicholson, F (1998) 'Management changes facing librarianship in Australia' Library Management vol 19 nº 5, pp333-338.

Schreiber, B and Shannon, J (2001) 'Developing library leaders for the 21st century' Journal of Library Administration vol 32 nº 3/4, pp35-57.

Sirianni, PM and Frey, BA (2001) 'Changing a culture: evaluation of a leadership development program at Mellon Financial Services' International Journal of Training and Development vol 5 nº 4, pp290-301.

Sogunro, OA (1997) 'Impact of training on leadership development: lessons from a leadership training program' Evaluation Review vol 21 nº 6, pp713-737, viewed 1 March 2003, Proquest database.

Sogunro, OA (1998) 'Leadership effectiveness and personality characteristics of group members' Journal of Leadership Studies vol 5 nº 3, viewed 26 April 2003, Expanded Academic ASAP database.

Stogdill, RM (1974) Handbook of leadership: a survey of theory and research Free Press, New York.

Summers, FW and Summers, L (1991) 'Library leadership 2000 and beyond: Snowbird Leadership Institute' Wilson Library Bulletin vol 66 nº 4, pp38-41.

Sutherland, A (2003) 'Recalling Aurora' Australian Library Journal vol 52 nº 2, pp116-117.

Weaver, B and Burger, L (1991) 'Library leaders for the 1990s' Wilson Library Bulletin vol 66 nº 4, pp35-37.

Biographical information

Kay Barney co-ordinated the 2003 Aurora Leadership Institute, which inspired her to complete her study for a Master of Library and Information Management at the University of Canberra. After a number of years working in journalism and economics, she is now at the Australian Bureau of Statistics Library in Canberra. Kay is also currently the associate editor of Australian Academic and Research Libraries. kaybarney@ozemail.com.au