How to make assessment more effective

Alternative assessment techniques can enhance student involvement, motivation or learning outcomes. Examples include peer assessment (Cheng & Warren, 1997; Sivan, 2000), portfolios (Tang & Biggs, 1998) and innovative feedback processes (Carless, 2002). (all cited in Keppel and Carless, 2006)

Sadler (cited in Carless, 2007) suggests that analysing high-quality exemplars is more effective than focusing on criteria and recounts the potent impact on his students of being exposed to outstanding assignments.



References:
Carless, D. (2007). Learning-oriented assessment: conceptual bases and practical implications. Innovations in Education and Teaching International 44.1, pp. 57–66.

Keppel, M. and Carless, D. (2006). Learning-oriented assessment: a technology-based case study. Assessment in Education, 13.2, pp. 179-191.

Learning Oriented Assessment

Assessment serves two main purposes: certification, which focuses principally on evaluating student achievement, and learning. When assessment is functioning efficiently, there should be substantial overlap between these two functions. However, the aim of assessment should be to strengthen the learning aspects of assessment; this can be achieved through either formative or summative assessments as long as a central focus is on engineering appropriate student learning. For formative assessments that learning purpose is an inherent characteristic, for summative assessments they develop productive student learning when the features described below are present.

Principle 1: Assessment tasks should be designed to stimulate sound learning practices amongst students
When assessment tasks embody the desired learning outcomes, students are primed for deep learning experiences by progressing towards these outcomes, akin to what Biggs (cited in Carless, 2007) describes as constructive alignment of objectives, content and assessment. ‘Alignment is evident when the articulation among learning objectives, content, instructional design, instructor expertise, technological affordances, and assessment strategies is as clear as possible’ (Reeves quoted in Keppel and Carless, 2006, p.181). The tasks should promote the kind of learning dispositions required of graduates and should mirror real-world applications of the subject matter. In addition, the tasks should aim to spread attention across a period of study, not lead to short-term bursts of sustained study for an end of module assignment; or in the case of examinations the memorising of material which is soon forgotten. As Macdonald suggests, ‘the design of assessment is critical in determining the direction of student effort’ (quoted in Keppel and Carless, 2006, p.182). Other characteristics of ‘assessment tasks as learning tasks’ include: a relationship between assessment tasks and real-world tasks; cooperative rather than competitive tasks, for example, through group work or project-based learning; and some degree of student choice in assessment tasks so as to facilitate student motivation and engagement.

Principle 2: Assessment should involve students actively in engaging with criteria, quality, their own and/or peers’ performance
Students should be involved in assessment so that they develop a better understanding of learning goals and engage more actively with criteria and standards. Within this strand (students’ involvement), it is included drafting criteria (Orsmond et al. cited in Carless, 2007); engaging with quality exemplars (Sadler cited in Carless, 2007); peer feedback (Liu & Carless cited in Carless, 2007) or peer assessment (Falchikov cited in Carless, 2007); and the development of self-evaluation skills or ‘evaluative expertise’ amongst students (Sadler cited in Carless, 2007). Through these activities, it is hoped that both the standards required and the transparency of the whole assessment processes can be enhanced.

Principle 3: Feedback should be timely and forward-looking so as to support current and future student learning
Students need to receive appropriate feedback which they can use to ‘feedforward’ into future work. Feedback in itself may not promote learning, unless students engage with it and act upon it (Gibbs & Simpson cited in Carless, 2007). Moreover, feedback should be less final and judgemental (Boud cited in Keppel and Carless, 2006) and more interactive and forward-looking (Carless cited in Keppel and Carless, 2006). Timeliness and promoting student engagement with feedback are thus key aspects. Although it is anticipated that the tutor would often provide feedback, peers can also be usefully deployed as givers of feedback (Falchikov cited in Carless, 2007).



References:
Carless, D. (2007). 'Learning-oriented assessment: conceptual bases and practical implications.' Innovations in Education and Teaching International 44.1, pp. 57–66.

Keppel, M. and Carless, D. (2006). Learning-oriented assessment: a technology-based case study. Assessment in Education, 13.2, pp. 179-191.

Assessment Challenges - handy summary

Assessment in higher education faces a number of challenges. Knight (2002) asserts that summative assessment is in ‘disarray’, for example, in terms of reliability or in judging the kind of complex learning to which higher education aspires. Formative assessment is also said to be in decline (Gibbs & Simpson, 2004) or failing to fulfil its potential (Knight & Yorke, 2003). All assessments lead to some kind of student learning (Boud, 1995), but a fundamental challenge is to stimulate the right kind of learning. Assessment tasks often distribute effort unevenly across a course (Gibbs & Simpson, 2004) and examinations are frequently critiqued for encouraging memorisation or surface approaches to learning (Ramsden, 2003). Assessment may also fail to support students in developing dispositions for lifelong learning, such as the ability to selfevaluate (Boud, 2000). Feedback is often ineffective, principally because it comes too late for students to use it productively (Carless, 2006) and it generally fails to include iterative cycles of feedback and revision that normally characterise academic writing (Taras, 2006).

One of the core problems is that assessment is about several things at once (Ramsden, 2003) or what Boud (2000) refers to as ‘double duty’. It is about grading and about learning; it is about evaluating student achievements and teaching them better; it is about standards and invokes comparisons between individuals; it communicates explicit and hidden messages. Assessment thus engenders tensions and compromises. Its multiple demands make its reform difficult to achieve, but in view of the centrality of assessment to the student experience, it is crucial that the area be handled well.

Copied from:
Carless, D. (2007). 'Learning-oriented assessment: conceptual bases and practical implications.' Innovations in Education and Teaching International 44.1, pp. 57–66.

Fundamental assessment principles to support student learning

1. Assessment tasks should be designed to ensure that students spend their study time in productive ways: tasks should encourage ‘time on task’ (eg, in and outside class), should lead to a more even distribution of study effort (over the timeline of the course), should engage students in deep rather than surface learning and should communicate clear and high expectations (Gibbs and Simpson cited in Nicol, 2007).
2. Assessment feedback should be of sufficient quantity, timely, it should focus on learning not just marks (emphasize the learning aspects of assessment, that is learning elements are emphasized more than measurement ones), it should be related to assessment criteria and be understandable, attended to and actually used by students to make improvements in their work (ibid).
   

References:
Nicol, D. (2007). 'Laying a foundation for lifelong learning: case studies of technology-supported assessment in large first year classes'. British Journal of Educational Technology. 38.4, pp. 668-678.

Curriculum as product: which problems arise for assessment

Education is most often seen as a technical exercise. Objectives are set, a plan drawn up, then applied, and the outcomes (products) measured.

Fundamental questions: a) What educational purposes should the school seek to attain?, b) How can we determine whether these purposes are being attained?

1) There are questions around the nature of objectives. This model is hot on measurability. It implies that behaviour can be objectively, mechanistically measured. There are obvious dangers here - there always has to be some uncertainty about what is being measured. We only have to reflect on questions of success in our work. It is often very difficult to judge what the impact of particular experiences has been. Sometimes it is years after the event that we come to appreciate something of what has happened. In order to measure, things have to be broken down into smaller and smaller units. The result, as many of you will have experienced, can be long lists of often trivial skills or competencies. This can lead to a focus in this approach to curriculum theory and practice on the parts rather than the whole; on the trivial, rather than the significant. It can lead to an approach to education and assessment which resembles a shopping list. When all the items are ticked, the person has passed the course or has learnt something. The role of overall judgment is somehow sidelined.
2) There is the problem of unanticipated results. The focus on pre-specified goals may lead both educators and learners to overlook learning that is occurring as a result of their interactions, but which is not listed as an objective.

I believe there is a tendency, recurrent enough to suggest that it may be endemic in the approach, for academics in education to use the objectives model as a stick with which to beat teachers. 'What are your objectives?' is more often asked in a tone of challenge than one of interested and helpful inquiry. The demand for objectives is a demand for justification rather than a description of ends... It is not about curriculum design, but rather an expression of irritation in the problems of accountability in education. (Stenhouse 1974: 77)

Copied from
Smith, M. K. (1996, 2000). Curriculum theory and practice. The encyclopedia of informal education. Retrieved: 11 January 2010. http://www.infed.org/biblio/b-curric.htm

Summative assessment in disarray

Summative assessment is being stretched to cover learning goals that resist robust, reliable and affordable summation; and that these assessment practices are shot through with contradictions such that the data they provide signify less than people tend to assume.

higher education institutions are generally expected to have learning goals that are far more extensive and complex than mastery of subject matter alone, and that they are being held to account for student achievement in terms of those goals.
public sector services are nowadays marked by low-trust management systems, when once there would have been a greater readiness to trust that good people engaged on worthwhile activities would learn the sorts of things that were intended. Assessment is supposed to supply evidence to bridge the trust gap.
quest for reliability tends to skew assessment towards the assessment of simple and unambiguous achievements, and considerations of cost add to the skew away from judgements of complex learning. To put it another way, high stakes assessments have trouble with the complex ambitions of higher education curricula and may actually impede them (Boud, 1995).
Assessment involves making assumptions about what exists, what it is like and how we might know about it. For example, if skills are nothing more than convenient terms for social practices that are decidedly situation-speci. c, and hence changeable (Holmes, 2001), then it will be frustrating to try to assess skills as if they were real, generalisable achievements.
there are objections to the claim that we have conscious access to all that we know, which raises obvious problems with trying to assess it. There is a strong line of thought which holds that much of what we know is tacit and distributed, and that much learning is non-formal.
The Limits of Reliability: fictional objects of assessment cannot be assessed with validity, and where validity is lacking, reliability is compromised.
The Stability of Assessment Judgements: Repeated observations are necessary before claiming that the observed behaviour is likely to be stable. This is important because if a higher education institution wishes to warrant achievement, then the warrant should be based on several assessors judging different instances of it. That
can hardly be done in a single module, but it might be done if there was an assessment plan covering a complete degree programme.
The Transferability of Achievement: The achievements that grades or degree classes signify may not be very transferable. Many psychologists hold transfer to be an achievement in its own right, not something that flows
freely and easily, except in familiar settings where speci. c transfer heuristics have been routinised (Anderson et al., 2000). Nor do scores and grades say anything about the learner’s ability to perform independently or in novel contexts. They may indicate a performance achieved with the help of plenty of scaffolding or with none. Even where two products are fairly awarded the same grade, there is a real difference between an achievement
where the task has been well defined, procedures for success have been laid down and plentiful guidance has been available, and another where no scaffolding has been provided. In other words, it is proper to doubt any assumption that warrants achievements that the learner can readily and independently transfer to fresh settings.
Limitations to Criteria-referencing: Although criteria-referenced assessment has many strengths, particularly compared to normreferencing, it is important to insist that benchmarks, speci. cations, criteria and learning outcomes do not and cannot make summative assessment reliable, may limit its validity and certainly compound its costs. For example, educational criteria are necessarily imprecise unless they refer to highly determined, even trivial achievements. Trainers may be able to develop and use precise-looking criteria, but educators work with fuzzy learning outcomes. Even ‘precise’ criteria are fuzzy to the extent: (i) that their meanings emerge in local communities of practice; and (ii) in the context of specific tasks (Wolf, 1997). Although criteria-referenced grading
may be good for student learning and equity in a community of practice, differences in the criteria used prevent, even impede, communication between communities; and make it impossible to be sure what any warrant means, since it is not possible to know what criteria have been used, what meanings have attached to them and how they have been used.
Assessment and Curriculum Skew: High stakes assessments, of the sorts that appear on transcripts and that lead to awards, have to be robust enough to stand up to legal challenge, so they tend to rest on assessments of things that people (often wrongly) believe can be judged reliably. This distorts the curriculum in two ways. First, what is subject to high stakes assessment gets serious attention and the rest does not. Secondly, achievements that are not warranted by high stakes assessment are neither recorded nor celebrated. The enacted curriculum becomes what high stakes judgements cover. Non-authentic assessments produce non-authentic curriculum, regardless of what the validated curriculum claims. There is a real danger that the frustrations of trying to assess
such accomplishments in reliable ways will lead to the use of national, content-free tests. Not only is their predictive validity in doubt (Sternberg, 1997), but if students concentrate on becoming test-smart, the tests’ consequential validity decreases because they actually distract students from the curriculum designed to teach those things that the tests claim to measure by proxy.

To compound matters, it should be understood that summative assessment may not be able to deliver what it is widely supposed to.

Copied from
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86.

Formative-Summative assessment: similarities and differences

While all assessments involve questions about validity, reliability, usefulness and cost, different assessment intentions lead to such different answers that there is a strong case for regarding feedout and feedback assessment as different systems. When the intention is to create feedback, learners need to be willing participants who disclose their uncertainties, errors and lacunae. Where feedout is the goal, disclosure is displaced by deception. Players in assessment games need to know which rules apply and there is little room to apply both sets to the same task. Not only do summative and formative assessments instantiate different rules of engagement, because they have different intentions, they also have different rules of evidence. Similarities are identified in the following list:
1. Assessors look for evidence of achievement.
2. Judgements are made about the match between evidence and criteria. In many cases conclusions are much less reliable, because the goodness of fit between evidence and criteria has to be inferred.
3. Judgement invokes communication. Some assessment judgements are relatively low stakes, may be fuzzy and exploratory and conversational in character, as in a discussion amongst peers or in dialogue with a tutor. In such cases there is often space on all sides for meanings to be negotiated and clarified. High stakes assessments are marked by unequal power relations between assessor and assessed, the format is not conversational and the judgement may be in a highly symbolic form, such as a number or letter grade, that is capable of multiple decodings. Understood as communications, summative and formative assessment are very different. The former is about conveying information, while the latter is more hermeneutic.
4. Judgements are economic processes. When people are assessing or being assessed, it is at the expense of doing other things. Stress (amongst teachers and learners) and dysfunctional behaviours (of which surface learning strategies can be one) are common consequences. And although formative assessment is intended to enrich the learning economy, summative assessment’s contribution to learning is less palpable.

Copied from
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86.

Assessment and learning

'The single, strongest influence on learning is surely the assessment procedures... even the form of an examination question or essay topics set can affect how students study … It is also important to remember that entrenched attitudes which support traditional methods of teaching and assessment are hard to change.' (Entwistle quoted in Knight, 2002, p. 276)

References:
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86