Australian Library and Information Association
home > publishing > alj > 54.3 > full.text >
 

The Australian Library Journal

High level learning by design: the nuts and bolts of assessment and evaluation in a Doctorate of Business Administration program

Suzanne Lipu, Allison Hill

Appendix A: Pre- and Post- Assessment Checklists

Includes raw data, selective graphic displays and the evaluation tool
Raw data

DBA Skills Checklist
  Pre-test Post-test
Question 1 2 3 4 5 1 2 3 4 5
Basic info research skills 1 2 3 4 5 1 2 3 4 5
Library website 2 0 0 0 9 0 0 0 0 11
Eresources 3 0 2 0 6 0 0 1 1 9
Dbase tutorials 1 1 3 3 3     1 1 9
Catalogue-books 0 2 2 3 4 0 0 0 2 9
Dictionary/encyclopedia 0 3 5 2 1 0 0 1 3 7
Journal titles 0 3 3 2 3 0 0 0 1 10
Theses 0 4 4 4 1 0 0 1 1 9
Limit-year   2   4 4       1 10
Limit-material type 1 3 2 1 3     2 1 8
Limit-publication type   3 1 6 2       2 9
Limit-location   1 5 3 2     1   9
Email catalogue records 1 3 2 2 3     1 1 9
Holds 2 1 3 3 2     1 1 9
Check borrowing record 1   2 3 3   2 1 2 7
Catalogue-access on/off campus 1 2   2 6       2 9
Catalogue-other libraries 2 4 4   1     2 1 8
JournalSearch vs catalogue 1 1 2 4 3       1 10
Access electronic journals 1 1 2 2 5         11
Identify relevant keywords 1   4 3 3         11
Construct a topic map 4 2 2 2         3 8
Identify 3 relevant dbases   2   4 1       3 8
Dbase-combine search terms   1 1 4 2     1 1 9
Dbase-truncate   2 3 3 2     1 1 9
Dbase-combine alternative terms   1 4 3 2     1 1 9
Dbase-author search   1 1 3 5     1 1 9
Dbase-limit search by year   1 1 4 4       2 9
Dbase-limit search by articles within a journal 1   5 2 2       1 8
Dbase-limit search by refereed journals 1 1 4 1 3       3 8
Dbase-print/save/email 1   3 1 5       2 9
Availability of additional databases 1 2 4 2 1     1 2 8
Dbases-search for theses 2 3 3 1 1     2 3 6
Dbase-search for newspapers 2 4 2 2     1 1 3 6
Advanced
Email alert 5 1 1 3   1   4 3 3
Web of Science 3 3   1 3     2 4 5
JCR 3 2 2 1 2     2 3 5
Where to publish 4   4 2       3 3 5
Company/industry via web 4 2 3 1       3 2 6
Company/industry via print 4 2 2 1       2 3 6
Statistical info - web/dbases 3 2 2 3 4     1 2 4
Statistical info - print 3 3 2 2 1     2 1 4
Statistical info - microform 3 4 3 1 1 1   3 2 1
Endnote
About EndNote 2 1 1 2 5       1 6
Differentiate between different sources info 3 1 1 5 1       1 6
Accurately cite sources in my work 3 1 1 4 2       1 6
Enter new references 3   1 2 5       1 6
Modify/edit references 3   1 5 2       1 6
Assign keywords 3 1 2 4 1       1 6
Use boolean searching 4 1 5 1       2 1 4
Search and download records from catalogue to EndNote library 4 2 2 3       1 1 5
Search and download records from dbases to EndNote library 5 2 1 3       1 1 5
Select appropriate bibliographic style 4 4   3     1   1 5
Edit bibliographic style 5 3 1 1 1       1 6
Insert citations into word 7 1 1 1 1     1 1 5
Format bibliography in word 6 2   2 1       3 4
Create independent, annotated bibliography 6 1 2 2       1 2 4
Other
Confident with information skills   2 5 3         6 1
Comfortable with information research skills 1 2 7 1         4 3
Confident in ability to identify own strengths and weaknesses in own information research skills   2 5 1 3       2 5

Graphic Display of selective raw data

Topic mapping

topic mapping

Catalogue

Tables 3-10 clearly show participants felt their catalogue skills had improved.

topic mapping

topic mapping

topic mapping

topic mapping

Databases

Graphs 11-18 illustrate participants' perception of new and consolidated knowledge about database searching.

topic mapping

topic mapping

topic mapping

topic mapping

In addition to developing skills to exploit the Library's catalogue and journal databases, participants were also exposed to a range of advanced journal techniques and additional resources, such as statistics and company information. See graphs 19-23 below.

Publishing

topic mapping

Company/Industry Information

topic mapping

Statistics via web/print

topic mapping

EndNote

A key assessment item for the subject was the development of an EndNote Library which made up 10 per cent of the final assessment mark. Participants indicated that this module presented a significant learning opportunity. See graphs 24-37 below for pre-test and post-test ratings.

topic mapping

topic mapping

topic mapping

topic mapping

topic mapping

topic mapping

topic mapping

Confidence in research skills

The skills gained from participation in the subject also had an impact on participants' confidence. The post-test data reveal a significant improvement in student confidence based on their self-assessment. See graph 39 below.

topic mapping

topic mapping

Checklist for Doctorate in Business Administration students

This checklist is designed for you to reflect on the wide range of Library resources and services available to help you with conducting research and keeping up to date with new developments in your research area. Another one will be given to you at the end of the subject. This will better enable us to help you and for you to review your development throughout the subject. In filling out this checklist please use the Likert rankings to reflect your answers. That is, circle 5 if you strongly agree, 4 if you agree, 3 if you are undecided, 2 if you disagree and 1 if you strongly disagree.

Basic information research skills
• I know the address of the library website 1 2 3 4 5
• I know that links to a wide range of subject related websites are listed on the relevant e-Resources pages for Commerce on the library website 1 2 3 4 5
• I know that web based database tutorials etc. are available on the library website 1 2 3 4 5
• When searching the Catalogue i know how to find ...          
   o books in my subject area 1 2 3 4 5
   o dictionaries and encyclopedias in my subject area 1 2 3 4 5
   o journal titles in my subject area 1 2 3 4 5
   o theses available in my subject area 1 2 3 4 5
• I know how to limit my search by ...          
   o year of publication 1 2 3 4 5
   o material type (eg videorecordings) 1 2 3 4 5
   o publication type (eg serials/journals) 1 2 3 4 5
   o location (eg reference) 1 2 3 4 5
• I know how to email catalogue records to myself 1 2 3 4 5
• I know how to place a hold request on a book out on loan to another person 1 2 3 4 5
• I know how to check my circulation (borrowing) record myself at any time 1 2 3 4 5
• I know how to access the catalogue from both on and off campus 1 2 3 4 5
• I know how to access other libraries' catalogues via the links on our library website 1 2 3 4 5
• I know the differences between JournalSearch & the library catalogue 1 2 3 4 5
• I know how to access the library's electronic journal titles using JournalSearch 1 2 3 4 5
• I know how to identify keywords and concepts relevant to my search needs 1 2 3 4 5
• I know how to construct a topic map 1 2 3 4 5
• I know how to identify at least three relevant databases in my particular subject area 1 2 3 4 5
• When searching a database I know that I can ...          
   o combine terms to narrow my results (advertising and internet) 1 2 3 4 5
   o truncate a search term to broaden my results (advertis*) 1 2 3 4 5
   o combine alternative terms to broaden my results (internet or world wide web) 1 2 3 4 5
   o search for items by a particular author 1 2 3 4 5
• I know that I can limit my search ...          
   o by year of publication 1 2 3 4 5
   o to articles within a particular journal title 1 2 3 4 5
   o to articles within refereed/peer reviewed journals (in certain databases) 1 2 3 4 5
   o I know that in most databases I can print, save and/or e-mail records to myself 1 2 3 4 5
• I know that I can obtain a separate handout which lists additional databases available for academics and research students 1 2 3 4 5
• I know how to search for theses using databases 1 2 3 4 5
• I know how to search for newspapers using databases 1 2 3 4 5
Advanced information research skills
 • I know how to set up an email alert in a database to help me keep up to date with new material published in my subject area 1 2 3 4 5
• I know how to use the citation indexes in the Web of Science database to follow through particular theories or authors 1 2 3 4 5
• I know how to use the Journal Citation Reports database to evaluate journal titles 1 2 3 4 5
• I know which other resources to used to determine which journals to publish in 1 2 3 4 5
• I know how to access the Library's range of company and industry information resources ...          
   o available via the web 1 2 3 4 5
   o held in print 1 2 3 4 5
• I know how to access the library's range of statistical information resources ...          
   o available via the web and on CD ROM database (eg AusStats, CLIB) 1 2 3 4 5
   o held in print 1 2 3 4 5
   o held on microfilm and microfiche (ie Australian historical statistics) 1 2 3 4 5
Endnote
• I know that EndNote is bibliographic software that enables storage, searching and management of up to 32 000 references 1 2 3 4 5
• I know how to differentiate between difference sources of information 1 2 3 4 5
• I know how to identify pertinent details to accurately cite sources I've used in my work 1 2 3 4 5
• I know how to enter a new reference into an EndNote library 1 2 3 4 5
• I know how to modify reference types to add or edit fields in an EndNote library 1 2 3 4 5
• I know how to assign appropriate keywords to an EndNote records to enable consistency and to obtain optimum search results 1 2 3 4 5
• I know how to apply Boolean concepts to searching for references in any EndNote library 1 2 3 4 5
• I know how to search and download records from the Library catalogue into an EndNote library using appropriate import filters 1 2 3 4 5
• I know how to search and download records from Library databases into an EndNote library using appropriate import filters 1 2 3 4 5
• I know how to select an appropriate bibliographic style to use in an EndNote library 1 2 3 4 5
• I know how to edit a bibliographic style in an EndNote library 1 2 3 4 5
• I know how to insert citations from an EndNote library into my word document 1 2 3 4 5
• I know how to format a bibliography from imported EndNote citations in word 1 2 3 4 5
• I know how to create an independent, annotated bibliography using EndNote 1 2 3 4 5
Other
• I feel confident in my information research skills 1 2 3 4 5
• I feel comfortable with my information research skills 1 2 3 4 5
• I feel confident in my ability to identify my own strengths and weaknesses in my information research skills 1 2 3 4 5

Allison Hill, Commerce Faculty librarian, ph 02 4221 3078, allison_hill@uow.edu.au
Suzanne Lipu, research training librarian, ph 02 4221 4350, suzannel@uow.edu.au
June 2003

Appendix B: Individual workshop evaluations

Includes results and evaluation tool
Results

Workshop one - introduction to searching, search strategies, the library web page, database introduction and postgraduate services

There was too much content covered in this workshop

Strongly agree Nil
Agree 9%
Undecided 36%
Disagree 36%
Strongly disagree 18%

It would have been better to have two presenters available for this workshop

Strongly agree 18%
Agree 27%
Undecided 18%
Disagree 18%
Strongly disagree 18%

The hands-on time in this workshop was sufficient

Strongly agree Nil
Agree 55%
Undecided 18%
Disagree 27%
Strongly disagree Nil

This workshop was relevant to my topic area

Strongly agree 45%
Agree 45%
Undecided Nil
Disagree Nil
Strongly disagree 9%

I will be able to use the skills and knowledge gained from this subject for my DBA course

Strongly agree 82%
Agree 9%
Undecided Nil
Disagree Nil
Strongly disagree 9%
Qualitative feedback:
'Well done'
'I thought this was well done to cover all the information in a short time. Maybe the volume of information was too much?'
'Thanks for giving me the opportunity out of class'

Workshop two - EndNote

There was too much content covered in this workshop

Strongly agree Nil
Agree 9%
Undecided 18%
Disagree 55%
Strongly disagree 18%

The hands-on time in this workshop was sufficient

Strongly agree Nil
Agree 82%
Undecided 9%
Disagree 9%
Strongly disagree Nil

This workshop was relevant to my topic area

Strongly agree 45%
Agree 45%
Undecided Nil
Disagree 9%
Strongly disagree Nil

I will be able to use the skills and knowledge gained from this subject for my DBA course

Strongly agree 82%
Agree 9%
Undecided Nil
Disagree Nil
Strongly disagree 9%
Qualitative feedback:
'Good workshop'
'Fantastic program'

Workshop three - advanced EndNote, targeting journals for publication, identifying and finding scholarly journals and research articles

There was too much content covered in this workshop

Strongly agree Nil
Agree Nil
Undecided 36%
Disagree 64%
Strongly disagree Nil

It would have been better to have two presenters available for this workshop

Strongly agree 9%
Agree 36%
Undecided 27%
Disagree 18%
Strongly disagree 9%

The hands-on time in this workshop was sufficient

Strongly agree Nil
Agree 64%
Undecided 27%
Disagree 9%
Strongly disagree Nil

This workshop was relevant to my topic area

Strongly agree 45%
Agree 45%
Undecided Nil
Disagree 9%
Strongly disagree Nil

I will be able to use the skills and knowledge gained from this subject for my DBA course

Strongly agree 82%
Agree 9%
Undecided Nil
Disagree Nil
Strongly disagree 9%
Qualitative feedback:
'Good workshop'
'Well done'

Workshop four - finding statistics, industry information including internet searching and evaluation, setting up journal alerts and conducting a literature review

There was too much content covered in this workshop

Strongly agree Nil
Agree Nil
Undecided 36%
Disagree 45%
Strongly disagree 18%

It would have been better to have two presenters available for this workshop

Strongly agree 9%
Agree 36%
Undecided 27%
Disagree 9%
Strongly disagree 18%

The hands-on time in this workshop was sufficient

Strongly agree 9%
Agree 64%
Undecided 27%
Disagree Nil
Strongly disagree Nil

This workshop was relevant to my topic area

Strongly agree 45%
Agree 36%
Undecided 9%
Disagree 9%
Strongly disagree Nil

I will be able to use the skills and knowledge gained from this subject for my DBA course

Strongly agree 82%
Agree Nil
Undecided 9%
Disagree Nil
Strongly disagree 9%
Qualitative feedback:
'Good workshop'
'Well done ladies'

Evaluation tool   DBA Workshops Evaluation

Please reflect on each of the individual workshops and then use the Likert rankings to indicate your impressions about each comment. That is, circle 5 if you strongly agree, 4 if you agree, 3 if you are undecided, 2 if you disagree and 1 if you strongly disagree. We have also provided a comments box for you to express any particular reflections about each workshop.

Workshop one - introduction to searching, search strategies, the library web page, database introduction, postgraduate services

• There was too much content covered in this workshop 1 2 3 4 5
• It would have been better to have two presenters available for this workshop 1 2 3 4 5
• The hands-on time in this workshop was sufficient 1 2 3 4 5
• This workshop was relevant to my topic area 1 2 3 4 5
• I will be able to use the skills and knowledge gained from this subject for my DBA course 1 2 3 4 5


Comments:


Workshop two - EndNote

• There was too much content covered in this workshop 1 2 3 4 5
• The hands-on time in this workshop was sufficient 1 2 3 4 5
• This workshop was relevant to my topic area 1 2 3 4 5
• I will be able to use the skills and knowledge gained from this subject for my DBA course 1 2 3 4 5


Comments:


Workshop three - advanced EndNote, targeting journals for publication, identifying and finding scholarly journals and research articles, citation indexes

• There was too much content covered in this workshop 1 2 3 4 5
• It would have been better to have two presenters available for this workshop 1 2 3 4 5
• The hands-on time in this workshop was sufficient 1 2 3 4 5
• This workshop was relevant to my topic area 1 2 3 4 5
• I will be able to use the skills and knowledge gained from this subject for my DBA course 1 2 3 4 5


Comments:


Workshop four - finding statistics, industry information including internet searching and evaluation, setting up journal alerts, literature reviews

• There was too much content covered in this workshop 1 2 3 4 5
• It would have been better to have two presenters available for this workshop 1 2 3 4 5
• The hands-on time in this workshop was sufficient 1 2 3 4 5
• This workshop was relevant to my topic area 1 2 3 4 5
• I will be able to use the skills and knowledge gained from this subject for my DBA course 1 2 3 4 5


Comments:


Thanks for your honest feedback. We look forward to seeing your presentations.

Suzanne and Allison

Appendix C: Subject evaluation

Includes results and evaluation tool
Results

1 How have you found the sequence of the program - for example in terms of content?
100 per cent of participants were positive about the sequence of the program
Comments included: 'Good: each session built on the last and I found the flow was logical and useful'; 'extremely relevant to research process'; 'Very good - helped my knowledge of research techniques'
2 What did you think about the timing of the program - in terms of length of workshops/timing of workshops?
75 per cent of participants thought the timing of the program was appropriate
17 per cent were indecisive
One participant commented: 'Suggest we may have a bit more time to "practise:"
3 What aspect/s of the program did you find the most useful?
100 per cent of participants said that learning EndNote was the most useful part of the program
33 per cent of participants said literature searching - including databases - was the most useful part
One participant noted that the Literature Review class was the most useful
One participant said that all the program was useful
One participant remarked on the support of the librarians: 'The opportunity for interaction with two very experienced librarians and the practical hands-on work with EndNote in the lab'
4 What aspect/s of the program did you find the least useful?
58 per cent of participants said that there was nothing they could identify as least useful
One participant said that finding statistics and legal information was the least useful
One participant said: 'More time needed for literature review to "refine" [topic]'
5 Was there any content are you would like more/less time on? If so, please explain.
42 per cent of participants were satisfied with the balance of content covered in the program
25 per cent of participants wanted more hands-on time, mainly for searching for resources on their topic area. One participant suggested that the course co-ordinator be available to assist when topic searching
33 per cent said they would like more time on EndNote
One participant commented that they would have liked: 'a more substantive focus on how to write up a literature review'
6 Did you feel you got enough one-on-one assistance? Could it be improved? If so please explain.
75 per cent felt that the assistance provided was sufficient
'Yes - Library staff are excellent - you are to be commended'
'Yes - I especially appreciate the offer of an individual appointment between sessions 5 and 6'
25 per cent felt that more one-on-one assistance would be helpful
'More one-on-one assistance needed'
7 Do you have any suggestions/recommendations for future programs?
58 per cent of participants had no suggestions or recommendations for future programs
One participant suggested that there be some more help on general information management
One participant said that the requirements of the literature review needed to be covered in more detail
Other comments included:
'Keep it going for all PhD students!'
'I think the subject outline could be improved so students have a better understanding of how the pieces of the various sessions fit together'
8 Any other comments:
One participant suggested that everyone receive a summary of each class member's research topic
Other comments included:
'I appreciate the willingness of the facilitators to help and their enthusiasm for the subject!'
'Really appreciated the input of Suzanne and Allison - very patient and knowledgeable and have opened up a raft of new knowledge to me with respect to research tools etc.'
'Great course, staff are excellent and very helpful'
'Great value'

Evaluation tool

Subject evaluation

  1. How have you found the sequence of the program - in terms of content?
  2. What did you think about the timing of the program - in terms of length of workshops/timing of workshops etc?
  3. What aspect/s of the program did you find the most useful?
  4. What aspect/s of the program did you find the least useful?
  5. Was there any content are you would like more/less time on? If so, please explain.
  6. Did you feel you got enough one-on-one assistance? Could it be improved? If so please explain.
  7. Do you have any suggestions/recommendations for future programs?
  8. Any other comments:

top
ALIA logo http://www.alia.org.au/publishing/alj/54.3/full.text/lipu.hill.appendix.html
© ALIA [ Feedback | site map | privacy ] sl,ah.sc 11:59pm 1 March 2010