Debate topics: August 2002

Welcome everyone to the 2002 INFOLIT discussion!

Thank you to colleagues who attended the 2002 Lifelong Learning Conference for being a catalyst for this discussion, in particular Christine Bruce, Judy Peacock, Shelia Webber, Bill Johnston and Ralph Catts.

My role in this discussion forum is to initiate debate and hopefully provoke responses! I've written a short reflection based on my own practice and reading that describes the distinction I see between assessment and evaluation and some of the assumptions behind assessment practice. This is not a scholarly paper (that will come later!!) so I haven't listed any references, however one book I highly recommend is Assessing open and distance learners by Chris Morgan and Med O'Reilly, 1999: Kogan Page. This book is great for anyone interested in assessment in any context, not just open and distance learning.

I've included an attachment of the same text as this e-mail just in case your e-mail does strange things with the formatting!

I'm interested in responses from list members about your context and the issues you face. Some questions for starters:

How do you assess information literacy learning?
Why do you assess information literacy learning?
Who marks the students' work?
What sort of feedback do you provide to students?
How do you evaluate information literacy teaching?
Why do you evaluate information literacy teaching?

Assessment and evaluation
I define assessment as describing or measuring learning outcomes. In other words, what has been learnt, or how much has been learnt. The way assessment is viewed may depend upon how one views learning: as a change in the way one sees the world, or as a change in the amount of knowledge/information one can acquire. My preference is for the former.

In the literature (especially US), there is some confusion about the nature and purpose of assessment. In many cases assessment seems to be used interchangeably with evaluation. I see a difference in the focus, purpose and intent of assessment and evaluation:

assessment is student focused while evaluation is teacher focused
the purpose of assessment is for students to receive feedback while the purpose of evaluation is for the teacher (and/or institution) to receive feedback
the intent of assessment is measuring/describing student learning, while the intent of evaluation is measuring/describing the effectiveness of teaching

Assessment can be used for evaluation as student learning outcomes can tell us a lot about teaching, but evaluation cannot be used for assessment.

I believe that an assessment task should be a learning activity with defined goals and learning outcomes. Further, there should always be a sound educational rationale for the use of any assessment instrument.

Why do we assess?
The purpose of assessment is to:

describe/measure learning
provide a structure for learning
provide opportunities for students and teachers to talk about learning
provide information for the evaluation of teaching
provide information for certification

Why do we evaluate?
The purpose of evaluation is to:

measure/describe the effectiveness of teaching, learning and assessment activities
measure/describe the effectiveness of teaching delivery
measure/describe the effectiveness of teaching method
measure/describe the effectiveness of curriculum and program design
examine the choice of content
provide opportunities for students, teachers, institutions and the community to talk about teaching and learning

We evaluate in order to improve the effectiveness of the teaching and learning experience. We want to improve learning outcomes. We want to know what works so we can do it better next time. We may want to diagnose the incoming level of students so that we can better design curriculum and programs to suit their background and experience. We may also need to evaluate to satisfy institutional, professional and government accountability requirements. Many of us in libraries need to justify the resources expended on information literacy education. We may have set performance indicators for our programs and ourselves.

Methods of evaluation include:

examining students' learning outcomes
asking students what they thought (questionnaires, focus groups, one-to-one interviews, informal conversations)
observations
debriefing with colleagues

How can we assess?
There are a myriad ways to assess including essays, tests, seminars, portfolios, journals, projects, reports, performances, theses, professional experiences, and observations. Those who assess could be the teacher, student and peers. Assessment can be informal, such as giving verbal feedback during class, or formal tasks intended to fulfil the requirements of the subject which may generate a mark or grade. In some disciplines such as medicine students may be assessed as pass or fail, and in competency-based training as competent or not yet competent.

Assessment can also be qualitative or quantitative. For example, three different self-assessment tasks could be:

students reflect on their learning in a journal or portfolio
students mark themselves against a set of criteria for a particular task
students respond to a list of statements filling in a self-rating scale where they agree or disagree to the statement

Different types of assessment may test different skills. Some test basic skills such as recall and comprehension, while others test critical thinking and problem solving. It is unlikely that assessment designed to test recall can test critical thinking, whereas it is possible that a task designed to test critical thinking may also test recall. An example of a 'rich' task that works at many different levels is the staged approach to an essay:

students submit an annotated bibliography of varied sources (eg books, journals, websites, primary, secondary) on their essay topic three or four weeks before the essay is due
students submit a draft two weeks before the essay is due
essay assessment criteria includes explicit information literacy outcomes

In order to complete the stages successfully students must have used lower order and higher order skills.

Issues in assessing information literacy in higher education
Looking through the library literature there are a number of issues for higher education institutions in assessing information literacy. Much of the assessment being reported seems to be run by the library independently of the curriculum. There is little reporting of assessment embedded in the curriculum. This does not mean that it is not happening, but just that people are not writing about it. My concern is that basing assessment in library programs keeps the responsibility for it in the library. If responsibility for information literacy assessment is kept in libraries, then the wider educational community may not take responsibility or may not realise that they need to take responsibility.

If information literacy remains library based, then it is likely that lower order skills are being assessed. It is difficult to do anything but measure basic skills in this context. It follows that if you ask a simple question, you get a simple answer. The authors of the paper accompanying the CAUL Information Literacy Standards state that 'recognition is needed that different levels of thinking skills are associated with various learning outcomes. Different instruments are essential to assess those outcomes' (p.4).

Many of the assessment instruments reported in the literature are multiple-choice items and self-rating items. But are these instruments measuring what they are intending to measure? Perhaps when intending to measure information literacy, we end up measuring information retrieval. Perhaps when intending to test an understanding of the catalogue we are actually testing knowledge of library jargon (eg OPAC).

Can we and should we assess information literacy?
Should information literacy be assessed? Can it be assessed? Why can't it be embedded into teaching, learning and assessment activities, not as something discrete, but as a holistic approach to education? Can it be separated from the context that initiated the information need?

Looking forward to the discussion,

Mandy

Dear INFOLITers,

Ralph Catts asked me to forward his response to a draft of my paper.

Here it is...

===============================================================
From: Ralph Catts <rcatts@metz.une.edu.au>
Subject: Re: Assessment debate

In my chapter in the book edited by Christine Bruce and Phil Candy I argue

'Information literacy is ... an essential capacity that students are required to demonstrate before they complete undergraduate education. ....Unless IL is an essential requirement for graduation, why should a student commit time to mastering the skills involved?' (Catts, 2000, 271). From the perspective of the University, unless students are required to demonstrate IL and have this assessed then how can the university certify that the graduate is information literate?

Therefore, I think that assessment of IL is not just for feedback, but is also for certification. This has to be done in such a way as to ensure that IL is demonstrated. Therefore, allocating 10 per cent of a grade for IL will not address the need. There has to be a compulsory assessment task on IL that must be satisfactorily addressed in order to pass.

To achieve that end in the undergraduate program I make it mandatory that students pass each assignment in the introductory unit and I include one assignment (AT LEAST) that is an assessment of IL. I then require use of IL skills as prerequisite knowledge for a subsequent unit.

I also teach at doctoral level (a professional doctorate) that has 4 coursework units followed by two years of professional practice research. In the unit I teach on 'professional practice: and reflection': I require students to prepare an annotated bibliography on a selected topic as a preliminary task. They have to gather current information from a range of sources, including on-line sources, assess the quality of the information, and summarise its import for their topic. This is a mandated assignment. They then use this information to inform their analysis of practice.

These are both examples of how teaching AND ASSESSING IL can both facilitate broader student learning of the content being taught, and provide the university with an assurance that the graduate has demonstrated IL.

Library based assessments of IL that are not tied to curriculum outcomes are useful for feedback purposes but are not SUFFICIENT to allow the university to conclude that IL has been achieved by every graduate.

IL is a complex phenomenon as it is described by Bruce (1997) or in the IL Standards (Caul, 2000). It requires comprehensive assessment and this should involve both satisfactory performance in searching, evaluating, and storing: (my compulsory assignment) plus effective and responsible use (my subsequent content based application assignment).

So my response to the final question in your paper is as follows. You ask 'why can't it be embedded into teaching learning and assessment activities'. My answer is that it can and it must! To achieve this end we need to convince lecturers and students that it is in their best interests. The simple solution is to show that IL students write assignments that are easier for lecturers to mark, and that IL students get higher grades (ie we appeal to self-interest). These issues are the focus of my current research.

Ralph

A a