The Australian Library Journal

High level learning by design: the nuts and bolts of assessment and evaluation in a Doctorate of Business Administration program

Suzanne Lipu, Allison Hill

Manuscript received November 2004

This is a refereed article

Introduction

Commerce is the largest faculty at the University of Wollongong; it supports postgraduate coursework, Masters and PhD programs. In 2003 work began on preparing a pilot program that was designed to position the Faculty at the top level of commerce-related research within Australia. It was called the Doctorate of Business Administration (DBA) and the plan was that it would be fully implemented into the Faculty's research program at a later stage. The pilot DBA program comprised four research subjects, four postgraduate level commerce subjects and the doctoral thesis itself. This paper is based on a subject called Advanced Business Specialisation Studies. It is the second of the four research subjects and requires the participants to find and organise relevant research literature in the preparation of their research proposal.

Right from early discussions about the pilot program and the Advanced Business Specialisation subject, it was felt that participants would require intensive information literacy instruction if they were going to produce top-class research. The Faculty realised that expertise in this area was going to be needed so the faculty librarian was approached to conduct a series of library-based sessions. She immediately saw the potential for full information literacy integration within the subject and sought the inclusion of the research training librarian in the program as well. A number of meetings were subsequently held with the Dean of Commerce and course co-ordinators to work out the content of the subject, negotiate assessment, assessment loading and the structure of course presentation. At this point both librarians were employed as lecturers with the realisation that their expertise and contribution were needed for the majority of the subject's content and delivery.

The pilot Doctorate of Business Administration (DBA) program

For the pilot DBA program, the Commerce Faculty wanted to attract successful business leaders who could become high-level researchers and whose work might provide significant status and recognition to the Faculty and the University. Twelve participants, from a variety of backgrounds, were recruited from the local area (see Table 1). The Faculty felt this would especially be of benefit to the participants, as their unique mix of ideas and viewpoints would be likely to create excellent research synergy.

With such a diverse range of backgrounds and research topics it was expected that the librarians would provide intensive support to fulfil different research needs and cater to varied levels of research skills. As most of the participants were working full-time, the classes were structured around five three-hour weekend workshops and one six-hour workshop. Times were flexible to meet participants' commitments. The two librarians determined the content of the subject - which was loosely based on previous Commerce postgraduate workshops, but geared towards developing research capabilities (Table 2).

Between classes participants had a considerable workload preparing for their research. The librarians and faculty staff monitored a joint e-mail account to answer questions, post information and follow issues through. Participants were expected to meet with the commerce faculty librarian or the research training librarian at least once to ensure their research was on the right track, to discuss all aspects of their topic and identify the availability of appropriate information. Although some participants arranged several appointments, most participants chose their appointments later in the timetable. Each participant was paired with an appropriate faculty-based mentor. They had regular meetings with their faculty mentors and course co-ordinators to provide additional support and monitor progress.

Both librarians felt that it was extremely important that quality assessment processes be used in this subject to encourage and foster effective student learning at the appropriate, sophisticated level required for research. As this was a pilot program evaluation was deemed critical, and feedback on the structure and organisation of the subject itself essential. The following sections outline the role and results of assessment and evaluation in the program.

Assessment in the DBA program

It is widely recognised that assessment and student learning go hand-in-hand. In a report to the Australian Universities Teaching Committee James, McInnis et al. (2002) from the Centre for the Study of Higher Education stated that:

The relationship between assessment and the overall quality of teaching and learning is often underestimated, yet assessment requirements and the clarity of assessment criteria significantly influence the effectiveness of student learning. Carefully designed assessment contributes directly to the way students approach their study and therefore contributes indirectly, but powerfully, to the quality of their learning (p11).

The American Association for Higher Education (2003) also has a set of Nine Principles of Good Practice for Assessing Student Learning. These include assessment that

reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time...clear, explicitly stated purposes...attention to outcomes but also and equally to the experiences that lead to those outcomes...

These important issues were considered in planning the subject assessment. The librarians ensured that the assessment and evaluation exercises were made explicit to the participants at all times. Exercises were aligned with the aims and objectives of the subject, integrated into content and paced throughout to allow for feedback and improvement. The librarians decided to assess participants on:

1. key skills that would enhance their research capabilities, and
2. pre and post-course information literacy skills in general

The aim was to not only 'test' participants on essential skills but to also to determine the effect of an intensive information literacy program on participants' knowledge and confidence in using their information literacy skills. To achieve this the participants had to complete a number of assigned tasks, skills checklists and evaluation questionnaires (Table 3).

The librarians decided that the most relevant tool for helping to assess information literacy in the DBA program was the first edition of the Australian Information Literacy Standards (2001) - the second, re-named edition not being released at that time. Lupton (2003) said that these are very useful in designing assessment because they are process-oriented and the statements could be used as learning outcomes. The librarians found this to be very accurate as they could easily relate specific statements from the Standards to the aims and objectives of the subject.

An example of the use of the Standards is shown below (Figure 1).

Like all assignments in this subject, assignment two built on skills and knowledge participants had previously developed, and prepared them for future learning and assessment tasks. They had already done a topic map, learnt how to use databases and import records into an EndNote library. In assignment two the students were advised to use records that would be relevant to their next assignment, the literature review.

The assignments received were of a high quality. Although the librarians did not mark assignment one it was evident the students had considered a wide range of issues relating to their research topic when constructing their topic map. Feedback from appointments indicated participants found the topic map a useful process for organising their thoughts. They used the topic map when completing their literature review, ensuring they covered all aspects of the research topic. The return of the EndNote disc and printout for assignment two meant many nights of careful marking for the librarians. Scores ranged from 7/10 to 9.5/10. All students received a comprehensive list of suggested improvements they could make to the 'style'of their EndNote bibliography. Faculty staff marked the literature review in assignment three. The marking of the participant's topic presentations for assignment four was a joint effort from the Faculty and the librarians. After consultation with the Faculty the librarians provided the marking criteria for all assessments. Use of the Standards to underpin the assessments facilitated order, structure and a clear division of tasks which greatly informed the marking process, providing a clear guideline for both markers and participants on what was expected from each assessment.

The role and results of evaluation in the DBA program

In addition to the assessment tasks the librarians conducted extensive evaluation throughout the subject. This was important because:

• this was a pilot program and evaluation was expected to provide feedback and recommendations for future courses in the Faculty, and
• it presented an opportunity to evaluate a comprehensive information literacy program within a research paradigm.

The librarians needed to determine, from the participants' perspective, if the subject helped them develop the key skills they needed to fulfil the objectives (and related assessments) of the subject. Skills checklists were used to determine the perceived information research skills of the participants. The checklists covered: use of basic and advanced information skills; EndNote skills, and perceived confidence levels in using research skills. Checklists were completed at the beginning and end of the workshops so results could be compared (Appendix A).

The research showed that the participants felt they had improved in their research skills abilities (refer to Appendix A). This is especially evident in skills students initially felt they had little experience with. For example, at the start of the subject participants had little or no knowledge in how to search/download catalogue and database records into Endnote. By the end of the subject 91 per cent of participants agreed that they did know how to use these skills. Individual workshop evaluations consistently showed that most participants felt that they would able to use the skills and knowledge gained from the subject in their DBA course (refer to Appendix B). From the librarians' perspective, the participants' ability to successfully complete assessments reinforced the research skills evaluations. Although the participants had not been asked if they felt the assessments were useful in assessing their level of research skills, the impressive quality of assessments handed in seemed to indicate that they had.

The last three questions of the Skills Checklists (Appendix A) focused on the perceived confidence of the participants in using their information research skills. Again the results clearly showed the majority of participants felt more confident and comfortable using their research skills. They also felt more confident in identifying their information research skills strengths and weaknesses. Cannon (2000) talks of evaluation of teaching as opposed to evaluation of learning. The evaluations described above focus on the latter. Cannon argues that evaluation of learning is the most desirable in terms of driving change in learning in universities through learning outcomes rather than what he describes as the prevalent 'tightly teacher-focused evaluations'. While the librarians agree in principle with his argument, they also wanted to evaluate the structure and organisation of the program because there is still validity and value in these factors both in terms of student learning and teaching. That is, they can help facilitate a conducive learning environment and they naturally provide some guidance about rolling out content - keeping in mind of course that flexibility and the ability to respond to student needs as they occur should also be of paramount concern. Individual workshop evaluations covered the content and structure of each workshop with room for comments (refer to Appendix B), while another evaluated the overall program/subject (Appendix C). Evaluations showed that the participants' response to the subject was overwhelmingly positive - especially in terms of the teaching support provided, relevancy of content and sequence of the subject. It also revealed some useful opportunities for improvement for future programs, namely in terms of ensuring that participants had sufficient time for hands-on experience with new concepts; and received clearer guidelines and support in constructing a literature review.

Outcomes from the pilot program

There were numerous outcomes of this pilot subject. Significant achievements included:

• the construction and delivery of a new, successful research preparation subject that would enhance the Faculty's professional doctoral program This was reflected in high quality assessment returns and positive feedback from the faculty and participants;
• increase in participants' information literacy skills and knowledge;
• the recognition of librarians' expertise in information literacy and, more significantly, the utilisation of this expertise in subject design as well as assessment and delivery;
• the two librarians found that collaborating with each other was an excellent professional development and training exercise (in terms of teamwork and synergy) and the culmination of their skills and knowledge resulted in high quality teaching;
• providing a practical exemplar for how academic librarians can use the Australian and New Zealand Information Literacy Framework: principles, standards and practice to develop an assessment plan.

In addition to these achievements various opportunities for improvement were identified for future programs. A selection of the most evident follows.

1. The sequence and content of information literacy integration throughout the DBA program should be repositioned in terms of point-of-need instruction.
2. The weighting of some assessment tasks need to be re-examined. For example, building the EndNote library required a number of skills (searching for relevant records and importing them into their Library; reformatting the style templates to reflect their Faculty's requirements and so on). The EndNote library was also meant to include records for their next assignment (the literature review). Given these factors and the amount of time it took participants to complete and librarians to mark, 10 per cent was probably not appropriate.
3. Participants required more time to be spent on literature reviews. This was reflected in qualitative feedback (refer to Appendices B and C).
4. Given the varying backgrounds of the high-achieving business leaders, the librarians felt that having two librarians present at each class provided the best opportunity to give individuals tailored assistance when it was most necessary, especially when it was evident that intervention at certain points would facilitate deeper learning and the ability to then grasp further concepts.

Finally, some outcomes have implications for the Library and Faculty. Firstly, the librarians' role as lecturers and subject designers was the result of significant discussion between library management and the Commerce Dean. The line between the normal duties of librarians (such as providing information literacy integration and classes) and academic duties (such as designing learning and assessment and marking) can blur and overlap. In this case, the librarians successfully negotiated academic status and remuneration largely because the classes were held outside normal working hours (on a Saturday morning) and heavier than 'normal' involvement in terms of design and delivery was being undertaken.

The future for similar programs however remains unclear. Unlike their counterparts in the United States, librarians working in the Australian higher education environment are not afforded academic status, despite the issue having been raised by numerous people in the field here and in the United Kingdom. (Revill 2001; Lupton 2002; Nimon 2002) It is the view of these authors that librarians are obviously equivalent to academics if they are involved in all aspects of the teaching and learning - such as in this program - and should be given the same status and pay as academics. After all, if they have appropriate qualifications and their experience and expertise in information literacy is being utilised in this way they are no different from academics also teaching in their area of expertise.

Conclusion

Overall, the opportunity to be true partners in this exciting subject benefited the participants, the Library, the librarians, the Commerce Faculty and others committed to information literacy in the higher education environment. The assessment and evaluation results, both qualitative and quantitative, demonstrated significant development of the participants' information literacy skills and knowledge - paving the way for them to become successful high-level researchers. The Library's profile and reputation was once again recognised and raised. Having the opportunity to develop and deliver a subject in full collaboration with Faculty staff, and indeed with each other, proved a positive professional development exercise for the librarians and encouraged the role of librarians as academics. The Faculty's ability to recruit and foster a significant cohort of regional business leaders in such a subject is testament to their reputation. The positive outcomes bode well for future programs, adding to the University's success. For others working in the information literacy environment these outcomes contributes a solid example of how the Australian and New Zealand Information Literacy Framework: principles, standards and practice can be applied in practice to design assessment for quality learning outcomes.

References

American Association for Higher Education (2003) 9 Principles of Good Practice for Assessing Student Learning, Washington Available online at: http://www.aahe.org/principl.html

Cannon, R (2000) 'Evaluating learning or evaluating teaching: is there a difference and does it matter?' in E Santhanam (ed.) Student feedback on teaching: reflections and projections: refereed proceedings of Teaching Evaluation Forum, 28-29 August 2000, Perth, Western Australia, pp 81-92. Crawley WA: University of Western Australia, Evaluation of Teaching Unit.

Council of Australian University Librarians (2001) Information Literacy Standards, Council of Australian University Librarians, Canberra. Available online at: http://www.caul.edu.au/caul-doc/InfoLitStandards2001.doc

James, R, McInnis, C and Devlin, M (2002) Assessing learning in Australian Universities, Centre for the Study for Higher Education, Melbourne. Available online at: http://www.cshe.unimelb.edu.au/assessinglearning/docs/AssessingLearning.pdf

Lupton, M (2002). 'The getting of wisdom: reflections of a teaching librarian' Australian Academic and Research Libraries 33(2): 75-85.

Lupton, M (2003) Assessment and Curriculum Alignment, Australian and New Zealand Institute of Information Literacy Symposium 2, Hamilton, New Zealand, December 2003. Available online at: http://www.anziil.org/resources/Alignment_of_IL_final.doc

Nimon, M (2002) 'Developing lifelong learners: controversy and the educative role of the academic librarian' Australian Academic and Research Libraries 33(1): 14-24.

Revill, D (2001) 'Future teaching roles for academic librarians: a review' New Library World 102(1168): 332-335.

Biographical information

Suzanne Lipu is the research training librarian at the University of Wollongong and an advisory member of the Research Group of ANZIIL (Australian and New Zealand Institute of Information Literacy). Suzanne has developed a comprehensive information literacy program for pre-service Education students, recently published in Information Literacy for Educators: Professional Knowledge for an Information Age (2004) and presented conference papers on information literacy and cultural diversity. Her research interests lie in information literacy, empowerment and internationalisation. She has previously worked in public and special libraries in Australia and held positions of college librarian/lecturer in Library Studies in Papua New Guinea Teacher's Colleges. Contact details: suzannel@uow.edu.au, ph 02 4221 4350.

Allison Hill is the Commerce Faculty librarian at the University of Wollongong. Allison has recently led a major updating project of the University's Information Literacies Introductory Program (a compulsory, non-credit point information literacy subject for all new undergraduates and postgraduate coursework students). In 2003 she co-presented a paper entitled 'Online introduction to information literacy: ticking that box or embedding that attribute?' at the ASCILITE (Australasian Society for Computers in Learning in Tertiary Education) conference. She has recently travelled to the University's campus in Dubai to establish an equivalent ILIP program for these offshore students in 2004. Contact details: allison@uow.edu.au, ph 02 422 13078.