AARL

Can an online tutorial pass the test for library instruction?

An evaluation and comparison of library skills instruction methods for first-year students at Deakin University

Marion Churkovich and Christine Oughtred

Abstract: An online, interactive tutorial, Smart Searcher, was introduced at Deakin University as part of the library's information skills program in late 2000. As liaison librarians responsible for library skills training we wanted to compare and evaluate this mode of instruction with our normal face-to-face delivery of library instruction. This study found that students with face-to-face instruction did, in fact, gain higher post-test mean scores than students completing the online tutorial. Also, students attending these library sessions felt more confident about their library skills than those in the online tutorial only session.

The Deakin University Library information literacy program offers training in a range of library and research skills to all levels of study. For first year students, liaison librarians at Deakin University aim to deliver library and research skills to 95% of first year on-campus students. This target reflects the library's strategic initiative in support of university objectives regarding the development of students' generic skills. The delivery of these programs normally takes place in tutorial times in negotiation with unit chairs and tutors of each course unit. Traditionally first year students are given a short tour of the library before an information skills session in the library computer laboratory. In the session students follow a demonstration of catalogue use and keyword searching, usually related to their essay topic, on an individual computer terminal. Time is then allowed for completion of an exercise sheet relating to these skills with the librarian present to answer questions. Students are encouraged to approach liaison librarians and information desk staff for queries.

An online tutorial was seen as a way of supporting the university's strategic objectives of offering online course delivery[1] and the library therefore decided to investigate the introduction of an interactive tutorial to provide consistency and flexibility in the delivery of information skills training. Using an online tutorial could also address problems with space and staff overload that sometimes arise with delivering information literacy classes for tutorial groups which are run simultaneously as well as consecutively. The online tutorial was also seen as a way of making training available to off-campus students.

Literature review

Much research in the area of distance education indicates there is no significant difference in student learning outcomes between online and lecture style instruction.[2] However, research into the evaluation and comparison of electronic programs to face-to-face teaching in the university library setting is less common. What research there is also indicates there is no significant difference between learning outcomes when using the different instructional methods.

Studies conducted in the United States, like our research, also compared library instruction via either a web-based tutorial or electronic workbook and lecture followed by hands-on computer experience. However, the research design differed from our study in each case and their electronic products were produced in-house whereas ours was customised commercially for Deakin University.

A study conducted by Germain, Jacobson and Kaczor[3] was similar to our proposed research in that it compared library instruction via a web-based tutorial with a lecture followed by hands-on computer experience. A total of 303 pre-tests and 284 post-tests were collected from first year undergraduates for evaluation. The instructional content of this study included 'the basics of using the library, its OPAC and entering a search in a database', and the time between testing varied between one and a half to six weeks. A librarian supervised those students undertaking the electronic tutorial. Results of the research suggested 'no difference in the effectiveness of the two types of instruction, web and live, based on the number of correct answers'. However, the authors did not plan to assign students to complete the tutorial independently.

Holman[4] also conducted similar research on the development of practical library skills by first year students, using either an online tutorial or attending a class, but again post-test times varied and the pre- and post-test questions were not identical. Students taking part in the librarian led class followed the demonstration on their own terminal but were not required to complete a follow-on exercise. Participants included 56 students who attended the class, 27 students who completed the tutorial and 42 students in a control group. Although students engaging in both instruction methods showed significant improvements over the control group, the tutorial group's post-test scores were not significantly different from those of the class group.

Kaplowitz and Contini[5] compared a computer-aided instruction (CAI) program with a lecture style instruction session for 423 first year biology students at UCLA. The CAI program, which was developed in-house and run over the 1994/5 academic year, used a concept-based approach. A pre-test was administered early in the semester and the post-test followed instruction and a homework assignment. Students' post-test responses seemed equivalent regardless of whether they had used the CAI or had attended the librarian's lecture and a decision was made to use the CAI program in future because it was less labour intensive.

Gutierrez, Wang & Herring[6] compared the performance of two groups of freshman college students assigned print and electronic workbooks respectively following a class lecture by a librarian. The researchers conducted a class lecture for each of two groups of approximately 65 students, eliminating the disadvantage of not having a 'human' instructor. No significant difference was found based on the format of the workbook. Their research suggested that frequency of library usage was the significant factor in improvement of information literacy skills rather than instructional format.

Research project

We felt further research was needed to investigate how the acquisition of basic research skills, appropriate for first year undergraduates, via the new Smart Searcher online tutorial compared with normal face-to-face instruction. We posed the question, 'Could first year students be trained successfully by an online tutorial without the presence of a librarian?'

Web-ezy tutorial

Deakin University Library purchased a customised version of the Web-ezy product produced by UNILINC, an organisation which coordinates library technologies in Australian higher education. This interactive program, renamed Smart Searcher for Deakin use, was customised to meet the needs of our users. Students are able to step through exercises that illustrate a range of information skills at their own pace. The eight modules within the tutorial cover skills such as catalogue, database and internet searching as well as the broader research skills of defining and understanding assignment topics. The tutorial is accessible to staff and students via links on the library's website and an offline version is available through the Deakin Learning Toolkit (a double CD ROM set produced for students containing information on the university's e-services and facilities as well as Deakin recommended software).

Smart Searcher also offers 'just in time' learning to students who need to have instruction now rather than wait for a class, and it is valuable reinforcement for concepts previously covered and possibly forgotten.

Methodology

First year sociology students at the Deakin University Geelong campus were asked to participate in this research as we liaise with staff from this area and regularly teach information skills to these students. The group, which consisted of some 200 students, is also made up of students from different faculties and a range of majors. One hundred and seventy-four students completed the instruction as well as the pre- and post-tests in their tutorial groups.

Test questions

The design of our questions was based on discussion with academic staff and research into methodological theory.[7] The pre-test had six demographic and attitudinal questions, 14 multiple-choice questions related to library skills and one open ended question asking for comments. The attitudinal questions related to willingness to ask for assistance and perceived competence in finding material in the library.

The multiple-choice questions were based on the Deakin University library skills training checklist at http://www.deakin.edu.au/library/checklist.html and reflected material covered in the Searching the Catalogue and Keyword Searching in the Catalogue modules of Smart Searcher. The basic information skills tested included:

• identifying and searching for reading list citations
• keyword searching
• knowledge of library resources.

These skills relate to Standard Two 'the information literate person accesses needed information effectively and efficiently' of the CAUL (Council of Australian University Librarians) Information Literacy Standards 2001[8] and specifically addresses outcome 2.2, 'the information literate person constructs and implements effectively designed search strategies'.

There was a subset of six questions, within the multiple-choice section, which tested the core skills of recognising the elements of a citation and applying that knowledge in searching for the item in the catalogue. The post-test had the same multiple-choice questions but also included an additional attitudinal question designed to measure satisfaction with the instruction.

The pre-test (see Appendix) was then circulated to academic staff for comment and trialed with six Year 12 secondary school students at the end of the 2000 school year.

Implementation

The research project was conducted during the second week of first semester 2001. Students stayed within their tutorial groups which were randomly assigned a method of instruction. The same class content, based on the core skills mentioned above, was covered in each group.

Group one - Tutorial A total of 68 students in four groups completed the Searching the Catalogue and Keyword Searching in the Catalogue modules of the Smart Searcher online tutorial without the assistance of a librarian. This tutorial, unlike the other two, was conducted in the general computer laboratories outside the library with the tutor in an observation role.

Group two - Mediated 45 students in two groups completed the same online modules of instruction, but with active assistance from a librarian who answered questions and corrected navigational errors.

Group three - Class 60 students from three groups had traditional face-to-face instruction, including a demonstration by a librarian, which they followed on their own terminal, and completed a paper exercise sheet.

All groups were administered the pre-test at the commencement of the session and the post-test at the conclusion. These were stapled together and colour coded by method of instruction for ease of data entry.

Results

SPSS (Version 10.0) software was used to input and analyse results. An Analysis of Covariance (ANOVA) was conducted and the minimum level of significance was set at 0.05. The measure used was the mean scores from the pre- and post-tests from each method of instruction. Scores were tabulated for the 14 multiple-choice questions that covered library skills. The results are displayed in the table below.

Table 1
Total pre- and post-test results by method of instruction

Our results indicate that students' post-test scores improved significantly as a result of library instruction, regardless of method. However, it is interesting to note that mean scores were not as high as we expected in either post-test. We suggest this reinforces the need for more comprehensive library instruction instead of relying solely on the one-off library class.

Surprisingly we found a significant difference between the methods of instruction, with the class group showing the greatest improvement between pre- and post-test scores. This finding differed from previous research, reviewed earlier in this article.

As can be seen in the graph below (Figure 1) the pre-test scores of the tutorial group were lower than those of the class and mediated groups, although overall these differences were not statistically significant. The tutorial group had no contact with a liaison librarian in their sessions, which were held outside the library.

Figure 1
Graph of pre-test and post-test results by method of instruction

Subset of questions

We had designed a subset of six questions, within the multiple choice section, related to identifying and searching for material likely to be on a first year student's reading list. Analysis of the subset proved to be of great interest. At this early stage of the semester, students obviously have trouble differentiating between journal articles, book chapters and book titles, which correlates with their lack of skill in locating the resources in our library. Following instruction, students in the class group improved significantly compared to the other two groups in this area. Mean scores for this subset are displayed in the table and graph below.

Table 2
Subset mean scores by method of instruction

Figure 2
Graph of subset mean scores by method of instruction

Library skills rating

Another significant finding from our study was that students who rated themselves poorly at finding library material in the pre-test did much better in a face-to-face class than those who attended the mediated or tutorial groups. Students were given the statement 'I know how to find library material for my course of study' and asked to respond with a choice from 'strongly agree', 'agree', 'undecided', 'disagree' and 'strongly disagree.' We divided the responses into three groups: those students who disagreed or strongly disagreed, those who strongly agreed or agreed, and the students who were undecided. We found a significant difference in all three groups, with the class group outperforming the mediated and tutorial participants, but this outcome was most pronounced for the group that gave themselves a poor rating. In this group the students attending the class session improved from a mean score of 6.65 to 10.11. Those in the tutorial group only improved from a pre-test mean score of 7.09 to a post-test mean of 8.09. The mediated groups' mean pre-test score was 7.13 and their post-test score was 8.25. The graph (Figure 3) below indicates the pre- and post-test mean scores of students who rated themselves poorly at finding library material.

Figure 3
Mean scores of students who rated themselves poorly in library skills

Confidence

A one-way analysis of variance was conducted to compare the post-training level of confidence for the different delivery methods (class, tutorial, mediated). In the post-test participants were asked to respond to the statement 'I feel more confident doing library research as a result of this tutorial session.' The overall level of confidence after the tutorial course was significantly less for the tutorial delivery, compared to the other methods. There was no significant difference between the class and mediated delivery methods in confidence after training.

Table 3
Confidence with library research as a result of instruction

This finding was also reflected in the comments section at end of the post-test. Almost 25% of the class group responded with personal comments of which 14 out of 15 were positive (the only negative comment was regarding the pace of the class). On the other hand, in the tutorial group only 13.2% responded and of those most (6 out of 9) were negative. These included comments such as 'frustrating', 'confusing' and 'difficult to follow'.

Discussion

Why did our results show a significant difference between the methods of instruction? When discussing library training it is pertinent to consider some principles of learning theory. These accepted principles of adult learning include appropriateness of training, motivation of trainees, reinforcement of skills, variety in presentation and actively involving participants.[10] We believe the class group did benefit from a more favourable learning environment.

In the class run by a librarian we included a short exercise which was relevant to their subject. This exercise helped reinforce the library skills taught in the session. This could also have contributed to the higher level of confidence in the class group. We felt the higher confidence levels of the class group following instruction were noteworthy as a positive attitude towards the library and its staff should lead to greater use and satisfaction in finding resources.

Even though the Smart Searcher tutorial does have a quick quiz at the end of each module, students were not actively supervised and we cannot know if any student completed that section.

David Brown,[11] Dean of the International Centre for Computer Enhanced Learning at Wake Forest University, has found that student learning appears to be highest when a course is neither all face-to-face nor all virtual but is instead a mixture of both. The results of our research would seem to support this view. The class group performed better in a session that combined an instructor-led demonstration, a practical online experience for each task and further reinforcement by completing a subject specific exercise sheet. This blend of presentation styles accommodates a variety of learning preferences and also allows the opportunity for questions and interaction with other students.

Deakin University's Smart Searcher is a commercially available customised web-based product whereas the other studies conducted research on electronic tutorials that the researchers had developed themselves. These in-house tutorials, which were produced for specific units, may have been more subject-based, thereby increasing motivation and post-test scores. Liaison librarians at Deakin University have raised the general nature of search examples within Smart Searcher as a potential limitation.[12]

We believe the above reasons, combined with some initial design flaws in navigating the pages of the module and certain inflexibility in its structure, influenced our results. The tutorial group was also isolated, to some extent, by completing the electronic modules in a computer lab outside the library and supervised by non-library staff. This may have had an influence on their motivation and attitude.

We also found it surprising that library skills were so poor in the pre-test. We suggest this reinforces the need for more comprehensive library instruction instead of relying solely on the one-off library class. The study completed by Gutierrez et al[13] found that those students who regularly used the library benefited most from library instruction. We agree with their premise that research skills require practice and a single library class is not sufficient. Although the class group in our study achieved the highest mean post-test score, there was room for improvement. Embedding research skills components in common or foundation units, and a consistent and comprehensive policy of addressing the needs of undergraduate students will become a priority for Deakin University Library in the future.

Conclusion

As a result of this research we intend to continue delivering face-to-face instruction for our subject-specific classes. However, Smart Searcher will be used as part of our First Year Introduction Program with a large number of students for basic catalogue skills training. This program is being run during Orientation Week and the first few weeks of semester.

The Smart Searcher tutorial content is to be updated shortly to reflect the new design of our library catalogue and redesigned website due to this research and other feedback. At this time a broader range of subject search examples will be incorporated into the content and, where possible, navigation problems will be addressed.

The results of our research indicate that contact with and instruction by a librarian is desirable for the best learning outcomes and confidence in development of information literacy skills. We attribute the success of the class group to flexible instruction, variety in presentation styles and reinforcement of concepts by a librarian.

Acknowledgments:

We would like to thank Dr Peter McCauley, our colleague at Deakin University Library and Dr Christine Armatas, senior lecturer, School of Psychology, Deakin University for their assistance in our data analysis.

A summary of this article appeared in Revelling in Reference 2001: RAISS Reference and Information Services Section Symposium Proceedings 12-14 October 2001.

Notes

1. B Lingham J Fletcher and G Henderson Online Tutorials: New Horizons in the Delivery of Flexible Online Training in Revelling in Reference 2001: RAISS Reference and Information Services Section Symposium Proceedings Melbourne Australian Library and Information Association 2001
2. T L Russell The No Significant Difference Phenomenon as Reported in 355 Research Reports, Summary and Papers: A Comparative Research Annotated Bibliography on Technology for Distance Education North Carolina State University 1999
3. C A Germain T E Jacobson and S A Kaczor 'A Comparison of the Effectiveness of Presentation Formats for Instruction: Teaching First-Year Students' College & Research Libraries vol 61 no 1 2000 pp65-72
4. L Holman 'A Comparison of Computer-Assisted Instruction and Classroom Bibliographic Instruction' Reference & User Services Quarterly vol 40 no 1 2000 pp53-65
5. J Kaplowitz and J Contini 'Computer-Assisted Instruction: Is It an Option for Bibliographic Instruction in Large Undergraduate Survey Classes?' College & Research Libraries no 59 1998 pp19-28
6. C Gutierrez J Wang and S D Herring 'A Comparison of an Electronic vs Print Workbook for Information Literacy Instruction'. Journal of Academic Librarianship vol 27 no 3 2001 pp208-220
7. D Shonrock Evaluating Library Instruction: Sample Questions, Forms, and Strategies For Practical Use Chicago American Library Association 1996
8. Council of Australian University Librarians Information Literacy Standards 1st ed Adelaide University of South Australia 2001
9. G Kroehnert Basic Training for Trainers: A Handbook for New Trainers Sydney McGraw-Hill 1990
10. D Brown 'Anticipating the Future in Universities and Colleges' in EDUCAUSE in Australasia 2001: The Power of 3, Bringing together Library, Teaching and Learning and Information Techonology Marriott Surfers Paradise Resort Gold Coast Queensland Martin Lack and Ass 2001
11. Ibid
12. Lingham, Fletcher and Henderson, op. cit.
13. Gutierrez, Wang and Herring, op. cit.

Appendix

Library Skills Tutorial Pre-Test
Thank you for completing this pre-test. You are participating in a research project involving library instruction at Deakin University. All information is anonymous and participation is voluntary.

You should tick one box only.

I agree to the use of these results in a published paper

  Yes o    No o

General

1. Have you had any library skills instruction in the past 2 years?

       Yes o    No o

2. How would you rate your computer skills?

       Very high o    Above average o    Average o    Below average o    Very low o

3. I feel comfortable asking for assistance in the library.

   Strongly agree o    Agree o    neutral/undecided o    Disagree o    Strongly disagree o

4. Learning to use the library resources efficiently is necessary for a student

   Strongly agree o    Agree o    neutral/undecided o    Disagree o    Strongly disagree o

The Catalogue

1. The following reference may appear in your reading list:
   Hall, C. 1982, 'The butcher, the baker, the candlestickmaker: The shop and the family in the industrial revolution' in The Changing Experience of Women, eds E. Whitelegg et al., Martin Robertson, Oxford.

   Is this reference for:
     o a book
     o a chapter in a book
     o a journal article
     o none of the above

2. What is the most effective search option to use if you had to locate the following book: Earle, l. Introduction to Australian society: a sociological overview. Sydney: Harcourt Brace Jovanovich

     o a keyword search
     o a title search
     o a subject search
     o an author search

3. What benefit is there to doing a keyword search on the Libraries' catalogue?

     o    The catalogue processes the search more quickly
     o    You don't need complete information about the item
     o    The search results only include items available to be checked out
     o    You may get thousands of items

4. By typing in human* in the catalogue you would find

     o     the plural of human
     o     an error message
     o     the closest match
     o     the various endings of human

5. What keyword would you choose for the following topic? What aspects of class produce differential health outcomes for individuals from different class backgrounds?

     o     individuals/class
     o    health outcomes
     o    class/health
     o    class background

6. Which call no. would enable you to find an item on the shelf?

     o    155
     o     368.09 How/Acr
     o     305.676
     o     516.7

7. At which campus is the following book available for loan?

   Author Pampel, Fred C
   Title Sociological lives and ideas : an introduction to the classical theorists
   New York : Worth Publishers, c2000

   Location	Call no.	Status
   DU Geelong	301.0922 Pam/Sla	Due 01-04-01
   DU Melb	301.0922 Pam/Sla	Available
   DU W'bool	301.0922 Pam/Sla	Due 07-04-01

     o    Geelong
     o     Melbourne
     o     Warrnambool
     o     Geelong and Warrnambool

8. Under what circumstances would you use hyperlinked subject headings in the catalogue?

     o     When you want to find other resources by the same author
     o    When you want to find out what resources the library is purchasing
     o    When you want to find other material on the same topic
     o    All of the above

9. You want to read the following article. How can you find it?

   Wacquant, L.1992, 'The Social Logic of Boxing in Black Chicago', Sociology of Sport Journal, vol. 9, pp221-54.

     o     Do an author search for Wacquant, L
     o    Do a title search for The Social Logic of Boxing in Black Chicago
     o    Do a title search for Sociology of Sport Journal
     o    Do a keyword search for boxing Chicago

10. The collection prefix for a book or journal article available for 2-hour loan is

      o     AVC
      o    SPTXT
      o    RSV
      o    GTC

Thank you for completing this pre-test.

Do you have any comments?

Marion Churkovich, liaison librarian, School of Social Inquiry, Deakin University, Geelong Victoria 3217.marionc@deakin.edu.au

Christine Oughtred, liaison librarian, School of Social Inquiry, Deakin University, Geelong Victoria 3217. co@deakin.edu.au.nospam (please remove the '.nospam' from the address).