Comparative fixed design

Comparative designs differs from cross-sectional in that a second, comparison group is selected in addition to the originally referred one.

While relational and comparative designs may seem very different, the difference is largely in the way the study is conducted. For example, in relational design the focus may be on relations between gender and attitude, while on comparative the focus could be on differences between the two groups (male/female).

References:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.159-160)

Relational fixed designs: cross sectional studies

Relational fixed designs measure the relationship between two or more variables
In a cross-sectional study, the forcus is on relationships between and among variables of a single group. The pattern of relationships between variables may be of interest in its own right, or there may be a concern for establishing causal links. The variables to be included in the study are those needed to provide answers to your research questions (independent-explanatory, dependent-outcome variables) and they are included because of their relevance to the research question. A rule of thumb sometimes proposed is a minimum of 15 participants per variable.
An issue is the homogeneity of the group. For instance, while you may not be interested in gender issues per se, may be males and females are affected by different variables and hence there will be increased variability on the outcome variable. A solution here is to analyse the 2 genders separately, i.e. to perform a subgroup analysis.

Category membership
The identification of membership of particular categories or groups can be difficult and complex. This may appear straightforward for a variable such as age. However, there are situations where even age can be problematic. Gender, as social construct, can raise category difficulties in particular cases.

Possible challenges
Establishing the statistical significance of the relationship between gender and attitude does not enable us to conclude that these variables are causally related. Nor does it, in itself, help in understanding what lies behind this relationship. We need to come up with plausible mechanisms and seek evidence for their existence. Further sub-division of the group allows to control statistically for the effects of such variables (gender-knowledge-concern with safety, see p.159), providing the relevant information has been collected.


References:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.154-109)

Fixed designs: some fundamental features

Fixed designs are theory driven. The only way in which we can, as a fixed design requires, specify in advance the variables to be included in our study and the exact procedures to be followed, is by having a reasonably well articulated theory of the phenomenon we are researching.

Causal models (like the one here) help to make clear the multiple causality of most things studied in social research. It emphasizes that while some variables may have a direct effect on others (e.g. that of assessment beliefs on assessment methods), others mhave an indirect effect (that of epistemological beliefs on assessment methods through assessment beliefs). Hard thinking to establish this kind of model before data collection is invaluable.

Fixed designs should always be piloted. Fixed designs are usually concerned with aggregates, with group properties and with general tendencies. Most non-experimental research also deals with averages and proportions. The relative weakness of fixed designs is that they cannot capture the subtleties and complexities of individual human behaviour. Even single case designs are limited to quantative measures of a single, or a small number, of simple behaviours. The advantage of fixed designs lies in their ability to transcend individual differences and identify patterns and processes which can be linked to social structrures and group organizational features.

Generally, researchers using fixed design methods remain at a greater physical and emotional distance from the study than those using flexible, qualitative methods.

The concept of 'variable' is part of the quantative worldview central to fixed design research. In this worldview, reality is conceptualized as consisting of properties of things, properties that can vary and thus be measured and compared. These properties may be seen as categorical (i.e. gender, nationality, teaching style) or as continuous (i.e. age, knowledge of some topic). It is very important to identify what variables are being studied, and to distinguish different types of variables (such as independent and depentdent variables).
independent variables: those which the experimenter manipulates.
dependent variables: those were we look for change.


issues of Validity, generalisability objectivity, credibility (p.100-109).

References:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.96-109)

A more refined topic and some thoughts about strategy selection

Which are the beliefs for assessment among ICT teachers of secondary-high schools, what assessment methods they employ (i.e. traditional paper-based/online) and how their assessment beliefs and methods are affected by with their implicit beliefs about knowledge and learning.

As Robson notes, the research questions help in determining the strategy. Therefore, taking into account that I am seeking to:

• Answer “which” questions (i.e. “which are the assessment beliefs of ICT teachers?”, “which are their epistemological beliefs?”), I should consider to deploy a non-epxerimental fixed strategy such as a survey
• Answer “how” and “what” questions (i.e. “what assessment methods ICT teachers prefer?”, “how their assessment beliefs and methods correlate with their implicit beliefs about knowledge and learning?”), I should also consider to deploy a flexible strategy.

However, there seems to be some problems in my topic concerning the implicit beliefs and their relation to assessment beliefs. How can I be so sure that there is indeed a mechanism, which is in operation? May be implicit beliefs does not affect assessment beliefs or methods, or that their relation is trivial while other mechanisms are more dominant. Therefore, it is clear the need to search the literature more thoroughly before proceeding to that question.

References:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.80, 91)

effort and learning

It's not that I'm so smart, it's just that I stay with problems longer.

Albert Einstein

Assessment challenges

Assessment carries negative overtones and perceptions
Assessment seems to have a negative image amongst many academics (e.g. concerns about the pernicious influences of assessment as accountability; a perception of assessment as a difficult and problematic field; heavy marking loads), while many of them simply equate assessment with grading or measurement (Carless,2007). Unfortunately, staffs’ views of assessment are sometimes relatively entrenched (Liu cited in Carless, 2007).


Low assessment awareness among teachers
Keppel and Carless (2006) suggest that teachers in training need to undergo some form of engagement with alternative approaches to assessment, so as trainee teachers can themselves experience their impact and raise their awareness of assessment as a learning process.

It is difficult to reform assessment in a curriculum
Attempts to introduce more diversified forms of assessment have shown that, in the school sector, assessment is the curriculum component that seems most difficult to reform (Morris et al., cited in Keppel and Carless, 2006). Potential barriers may rise from teachers’ existing beliefs (when they are not congruent with the assessment elements being promoted) (Carless cited in Keppel and Carless, 2006), institutional quality assurance guidelines (Carless, 2007), etc.

assessment fails to develop higher order outcomes
Students are oriented towards assessment which values rote learning and the accumulation of marks rather than deep understanding. (Watkins cited in Keppel and Carless, 2006). Indeed, examinations or one-off end of module assignments often fail to promote learning dispositions (Keppel and Carless, 2006).

Assessment feedback usually comes too late
It is rather common for students to receive most of their feedback after a module is completed. Unfortunately, then there is minimal possibility of it being acted upon or use it effectively.


Individual contribution in a group project and peer grading
There seems to be no consensus among learners about individual grading in a group project.
In a case-study conducted by Keppel and Carless (2006):
35% of students were unsure about whether or not their individual contribution to a group project should be graded while 57.5% suggested that their individual contribution to the project should be graded.
Contrarily, it seems that most (72.5%) of the learners feel that it would be acceptable to grade other members of their group in relation to their contribution to the group project.


Students’ conceptions about feedback may lower its efficiency
Keppel and Carless (2006) suggest that students may view teacher feedback as the only valid
form of feedback, discounting peer feedback, or may think of feedback as being individual formal written comments rather than verbal advice within teaching sessions.


Assessment preferences of pragmatic students
Although alternative and innovative methods of assessment can lead to productive learning experiences, a pragmatic student may prefer a more traditional one-off end of module assignment which involves less time and effort. These methods may indeed impose higher workload and thus they require from students to buy into the approach. (Keppel and Carless, 2006)

grades a tool for teachers at the expense of students and education
Grades offer teachers a convenient device for allaying their anxieties about their own abilities by shifting them onto their students, through an endless round of tests, examinations, and evaluations. Grades get teachers off the hook; they preserve professorial authority and are indifferent to professorial incompetence. Bad-faith protestations about administration requirements can mask the fact that grades serve the teacher at the expense of the students, and at the sacrifice of education. (copied from Noble)

End of year examinations are poor predictors of subsequent performance
Examinations are very poor predictors of any subsequent performance, such as success at work. A review of 150 studies of the relationship between exam results and a wide range of adult achievement found the relationship to be, at best, slight (Baird cited in Gibbs and Simpson, 2004-5). Contrarily, coursework marks are a better predictor of long term learning of course content than are exams. Presumably the kind of learning that coursework involves has long term consequences while the kind of learning involved in revision for exams does not (Gibbs and Simpson, 2004-5).

Grades have a negative impact on students’ self-efficacy
A grade is likely to be perceived by the student as indicating their personal ability or worth as a person as it is usually ‘norm-referenced’ and tells you, primarily, where you stand in relation to others. A poor grade may damage a student’s ‘self-efficacy’, or sense of ability to be effective. Moreover, In the absence of marks it has been reported that students read feedback much more carefully (Black & Wiliam cited in (Gibbs and Simpson, 2004-5) and use it to guide their learning (Gibbs and Simpson, 2004-5).

Assessment is judgement and it might abuse students
Learning is an act which necessarily leaves the student vulnerable while teachers usually judge too much and too powerfully, not realising the extent to which students experience their power over them (Boud, 1995). Teachers ‘must inevitably look at the profile of assessment as students see it’ (Boud, 1995, p.5). One who conducts an assessment to his trainees should undergo a similar experience, now and then, in order to keep in mind the stressful feelings it often induces.


References:

Boud, D. (1995). Assessment and learning: contradictory or complementary. In: Knight, P. ed. Assessment for Learning in Higher Education. London: Kogan Page/SEDA. pp. 35-48.

Carless, D. (2007). Learning-oriented assessment: conceptual bases and practical implications. Innovations in Education and Teaching International 44.1, pp. 57–66.

Gibbs, G. and Simpson, C. (2004-5). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education, 1, pp.3-31.

Keppel, M. and Carless, D. (2006). Learning-oriented assessment: a technology-based case study. Assessment in Education, 13.2, pp. 179-191.

Noble, D. May 2007: Giving Up the Grade. Ending grading frees academia, reintroduces education.
http://www.policyalternatives.ca/publications/monitor/may-2007-giving-grade

How to make assessment more effective

Alternative assessment techniques can enhance student involvement, motivation or learning outcomes. Examples include peer assessment (Cheng & Warren, 1997; Sivan, 2000), portfolios (Tang & Biggs, 1998) and innovative feedback processes (Carless, 2002). (all cited in Keppel and Carless, 2006)

Sadler (cited in Carless, 2007) suggests that analysing high-quality exemplars is more effective than focusing on criteria and recounts the potent impact on his students of being exposed to outstanding assignments.



References:
Carless, D. (2007). Learning-oriented assessment: conceptual bases and practical implications. Innovations in Education and Teaching International 44.1, pp. 57–66.

Keppel, M. and Carless, D. (2006). Learning-oriented assessment: a technology-based case study. Assessment in Education, 13.2, pp. 179-191.

Learning Oriented Assessment

Assessment serves two main purposes: certification, which focuses principally on evaluating student achievement, and learning. When assessment is functioning efficiently, there should be substantial overlap between these two functions. However, the aim of assessment should be to strengthen the learning aspects of assessment; this can be achieved through either formative or summative assessments as long as a central focus is on engineering appropriate student learning. For formative assessments that learning purpose is an inherent characteristic, for summative assessments they develop productive student learning when the features described below are present.

Principle 1: Assessment tasks should be designed to stimulate sound learning practices amongst students
When assessment tasks embody the desired learning outcomes, students are primed for deep learning experiences by progressing towards these outcomes, akin to what Biggs (cited in Carless, 2007) describes as constructive alignment of objectives, content and assessment. ‘Alignment is evident when the articulation among learning objectives, content, instructional design, instructor expertise, technological affordances, and assessment strategies is as clear as possible’ (Reeves quoted in Keppel and Carless, 2006, p.181). The tasks should promote the kind of learning dispositions required of graduates and should mirror real-world applications of the subject matter. In addition, the tasks should aim to spread attention across a period of study, not lead to short-term bursts of sustained study for an end of module assignment; or in the case of examinations the memorising of material which is soon forgotten. As Macdonald suggests, ‘the design of assessment is critical in determining the direction of student effort’ (quoted in Keppel and Carless, 2006, p.182). Other characteristics of ‘assessment tasks as learning tasks’ include: a relationship between assessment tasks and real-world tasks; cooperative rather than competitive tasks, for example, through group work or project-based learning; and some degree of student choice in assessment tasks so as to facilitate student motivation and engagement.

Principle 2: Assessment should involve students actively in engaging with criteria, quality, their own and/or peers’ performance
Students should be involved in assessment so that they develop a better understanding of learning goals and engage more actively with criteria and standards. Within this strand (students’ involvement), it is included drafting criteria (Orsmond et al. cited in Carless, 2007); engaging with quality exemplars (Sadler cited in Carless, 2007); peer feedback (Liu & Carless cited in Carless, 2007) or peer assessment (Falchikov cited in Carless, 2007); and the development of self-evaluation skills or ‘evaluative expertise’ amongst students (Sadler cited in Carless, 2007). Through these activities, it is hoped that both the standards required and the transparency of the whole assessment processes can be enhanced.

Principle 3: Feedback should be timely and forward-looking so as to support current and future student learning
Students need to receive appropriate feedback which they can use to ‘feedforward’ into future work. Feedback in itself may not promote learning, unless students engage with it and act upon it (Gibbs & Simpson cited in Carless, 2007). Moreover, feedback should be less final and judgemental (Boud cited in Keppel and Carless, 2006) and more interactive and forward-looking (Carless cited in Keppel and Carless, 2006). Timeliness and promoting student engagement with feedback are thus key aspects. Although it is anticipated that the tutor would often provide feedback, peers can also be usefully deployed as givers of feedback (Falchikov cited in Carless, 2007).



References:
Carless, D. (2007). 'Learning-oriented assessment: conceptual bases and practical implications.' Innovations in Education and Teaching International 44.1, pp. 57–66.

Keppel, M. and Carless, D. (2006). Learning-oriented assessment: a technology-based case study. Assessment in Education, 13.2, pp. 179-191.

Different interpretations and conceptions of formative assessment hinder its development

Formative assessment is more complex than it appears at first sight (Yorke cited in Carless, 2007). Formative assessment can be viewed as planned and structured or as informal and ad hoc, what Bell and Cowie refer to as interactive formative assessment (Bell and Cowie cited in Carless, 2007). This phenomenon may have practical impediments to formative assessment. Those that favour a formal and planned approach to formative assessment are inclined to believe that teachers do not have the time to conduct formative assessments, while those with the more constructivist interactive orientation tend to believe that formative assessment is just a fundamental part of good teaching (Carless, 2007). A further concern was students’ responses to formative assessment and in particular whether it may unduly increase their workload burden. Instrumental students may interpret some types of formative assessment as extra work for no marks or regard them as non-essential (Yorke cited in Carless, 2007).



References:
Carless, D. (2007). 'Learning-oriented assessment: conceptual bases and practical implications.' Innovations in Education and Teaching International 44.1, pp. 57–66.

Assessment Challenges - handy summary

Assessment in higher education faces a number of challenges. Knight (2002) asserts that summative assessment is in ‘disarray’, for example, in terms of reliability or in judging the kind of complex learning to which higher education aspires. Formative assessment is also said to be in decline (Gibbs & Simpson, 2004) or failing to fulfil its potential (Knight & Yorke, 2003). All assessments lead to some kind of student learning (Boud, 1995), but a fundamental challenge is to stimulate the right kind of learning. Assessment tasks often distribute effort unevenly across a course (Gibbs & Simpson, 2004) and examinations are frequently critiqued for encouraging memorisation or surface approaches to learning (Ramsden, 2003). Assessment may also fail to support students in developing dispositions for lifelong learning, such as the ability to selfevaluate (Boud, 2000). Feedback is often ineffective, principally because it comes too late for students to use it productively (Carless, 2006) and it generally fails to include iterative cycles of feedback and revision that normally characterise academic writing (Taras, 2006).

One of the core problems is that assessment is about several things at once (Ramsden, 2003) or what Boud (2000) refers to as ‘double duty’. It is about grading and about learning; it is about evaluating student achievements and teaching them better; it is about standards and invokes comparisons between individuals; it communicates explicit and hidden messages. Assessment thus engenders tensions and compromises. Its multiple demands make its reform difficult to achieve, but in view of the centrality of assessment to the student experience, it is crucial that the area be handled well.

Copied from:
Carless, D. (2007). 'Learning-oriented assessment: conceptual bases and practical implications.' Innovations in Education and Teaching International 44.1, pp. 57–66.

Fundamental assessment principles to support student learning

1. Assessment tasks should be designed to ensure that students spend their study time in productive ways: tasks should encourage ‘time on task’ (eg, in and outside class), should lead to a more even distribution of study effort (over the timeline of the course), should engage students in deep rather than surface learning and should communicate clear and high expectations (Gibbs and Simpson cited in Nicol, 2007).
2. Assessment feedback should be of sufficient quantity, timely, it should focus on learning not just marks (emphasize the learning aspects of assessment, that is learning elements are emphasized more than measurement ones), it should be related to assessment criteria and be understandable, attended to and actually used by students to make improvements in their work (ibid).
   

References:
Nicol, D. (2007). 'Laying a foundation for lifelong learning: case studies of technology-supported assessment in large first year classes'. British Journal of Educational Technology. 38.4, pp. 668-678.

Why formative assessment is important

• the number of opportunities available for formative assessment and feedback is an important variable in non-completion by students in the early years of study even though a direct causal connection has been difficult to prove (Yorke cited in Nicol, 2007).
• Where students are uncertain about their ability to succeed, formative assessment and feedback is of particular significance (Yorke and Longden cited in Nicol, 2007).
• Increased emphasis on summative assessment at the expense of formative assessment results in an excessive concentration by students on getting good marks and playing the assessment game rather than on focusing their effort on deep and lasting learning. These changes have also a subsequent negative impact on the students’ sense of self and on their motivation and self-confidence (Nicol, 2007).

Copied from
Nicol, D. (2007). 'Laying a foundation for lifelong learning: case studies of technology-supported assessment in large first year classes'. British Journal of Educational Technology. 38.4, pp. 668-678.

A framework for Research Design

Purposes
Theory
Research Questions
Methods
Sampling strategy


see: Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.81)

Approaches in course design, view of knowledge, purpose of assessment and methods

The summary table offers some interesting connections between course design approaches and view of knowledge-assessment purposes and methods.View at:

http://holyroodpark.pbworks.com/CD10_Approaches-Summary-Table
or saved as Toohey table.

Toohey (1999) a must reading about beliefs, values and ideologies in course design.
'The most accurate picture of the educational values and beliefs inherent in a course is allocated to different topics and learning activities and in the nature of the assessment.' (p.49)

Curriculum as product: which problems arise for assessment

Education is most often seen as a technical exercise. Objectives are set, a plan drawn up, then applied, and the outcomes (products) measured.

Fundamental questions: a) What educational purposes should the school seek to attain?, b) How can we determine whether these purposes are being attained?

1) There are questions around the nature of objectives. This model is hot on measurability. It implies that behaviour can be objectively, mechanistically measured. There are obvious dangers here - there always has to be some uncertainty about what is being measured. We only have to reflect on questions of success in our work. It is often very difficult to judge what the impact of particular experiences has been. Sometimes it is years after the event that we come to appreciate something of what has happened. In order to measure, things have to be broken down into smaller and smaller units. The result, as many of you will have experienced, can be long lists of often trivial skills or competencies. This can lead to a focus in this approach to curriculum theory and practice on the parts rather than the whole; on the trivial, rather than the significant. It can lead to an approach to education and assessment which resembles a shopping list. When all the items are ticked, the person has passed the course or has learnt something. The role of overall judgment is somehow sidelined.
2) There is the problem of unanticipated results. The focus on pre-specified goals may lead both educators and learners to overlook learning that is occurring as a result of their interactions, but which is not listed as an objective.

I believe there is a tendency, recurrent enough to suggest that it may be endemic in the approach, for academics in education to use the objectives model as a stick with which to beat teachers. 'What are your objectives?' is more often asked in a tone of challenge than one of interested and helpful inquiry. The demand for objectives is a demand for justification rather than a description of ends... It is not about curriculum design, but rather an expression of irritation in the problems of accountability in education. (Stenhouse 1974: 77)

Copied from
Smith, M. K. (1996, 2000). Curriculum theory and practice. The encyclopedia of informal education. Retrieved: 11 January 2010. http://www.infed.org/biblio/b-curric.htm

Summative assessment in disarray

Summative assessment is being stretched to cover learning goals that resist robust, reliable and affordable summation; and that these assessment practices are shot through with contradictions such that the data they provide signify less than people tend to assume.

higher education institutions are generally expected to have learning goals that are far more extensive and complex than mastery of subject matter alone, and that they are being held to account for student achievement in terms of those goals.
public sector services are nowadays marked by low-trust management systems, when once there would have been a greater readiness to trust that good people engaged on worthwhile activities would learn the sorts of things that were intended. Assessment is supposed to supply evidence to bridge the trust gap.
quest for reliability tends to skew assessment towards the assessment of simple and unambiguous achievements, and considerations of cost add to the skew away from judgements of complex learning. To put it another way, high stakes assessments have trouble with the complex ambitions of higher education curricula and may actually impede them (Boud, 1995).
Assessment involves making assumptions about what exists, what it is like and how we might know about it. For example, if skills are nothing more than convenient terms for social practices that are decidedly situation-speci. c, and hence changeable (Holmes, 2001), then it will be frustrating to try to assess skills as if they were real, generalisable achievements.
there are objections to the claim that we have conscious access to all that we know, which raises obvious problems with trying to assess it. There is a strong line of thought which holds that much of what we know is tacit and distributed, and that much learning is non-formal.
The Limits of Reliability: fictional objects of assessment cannot be assessed with validity, and where validity is lacking, reliability is compromised.
The Stability of Assessment Judgements: Repeated observations are necessary before claiming that the observed behaviour is likely to be stable. This is important because if a higher education institution wishes to warrant achievement, then the warrant should be based on several assessors judging different instances of it. That
can hardly be done in a single module, but it might be done if there was an assessment plan covering a complete degree programme.
The Transferability of Achievement: The achievements that grades or degree classes signify may not be very transferable. Many psychologists hold transfer to be an achievement in its own right, not something that flows
freely and easily, except in familiar settings where speci. c transfer heuristics have been routinised (Anderson et al., 2000). Nor do scores and grades say anything about the learner’s ability to perform independently or in novel contexts. They may indicate a performance achieved with the help of plenty of scaffolding or with none. Even where two products are fairly awarded the same grade, there is a real difference between an achievement
where the task has been well defined, procedures for success have been laid down and plentiful guidance has been available, and another where no scaffolding has been provided. In other words, it is proper to doubt any assumption that warrants achievements that the learner can readily and independently transfer to fresh settings.
Limitations to Criteria-referencing: Although criteria-referenced assessment has many strengths, particularly compared to normreferencing, it is important to insist that benchmarks, speci. cations, criteria and learning outcomes do not and cannot make summative assessment reliable, may limit its validity and certainly compound its costs. For example, educational criteria are necessarily imprecise unless they refer to highly determined, even trivial achievements. Trainers may be able to develop and use precise-looking criteria, but educators work with fuzzy learning outcomes. Even ‘precise’ criteria are fuzzy to the extent: (i) that their meanings emerge in local communities of practice; and (ii) in the context of specific tasks (Wolf, 1997). Although criteria-referenced grading
may be good for student learning and equity in a community of practice, differences in the criteria used prevent, even impede, communication between communities; and make it impossible to be sure what any warrant means, since it is not possible to know what criteria have been used, what meanings have attached to them and how they have been used.
Assessment and Curriculum Skew: High stakes assessments, of the sorts that appear on transcripts and that lead to awards, have to be robust enough to stand up to legal challenge, so they tend to rest on assessments of things that people (often wrongly) believe can be judged reliably. This distorts the curriculum in two ways. First, what is subject to high stakes assessment gets serious attention and the rest does not. Secondly, achievements that are not warranted by high stakes assessment are neither recorded nor celebrated. The enacted curriculum becomes what high stakes judgements cover. Non-authentic assessments produce non-authentic curriculum, regardless of what the validated curriculum claims. There is a real danger that the frustrations of trying to assess
such accomplishments in reliable ways will lead to the use of national, content-free tests. Not only is their predictive validity in doubt (Sternberg, 1997), but if students concentrate on becoming test-smart, the tests’ consequential validity decreases because they actually distract students from the curriculum designed to teach those things that the tests claim to measure by proxy.

To compound matters, it should be understood that summative assessment may not be able to deliver what it is widely supposed to.

Copied from
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86.

Formative-Summative assessment: similarities and differences

While all assessments involve questions about validity, reliability, usefulness and cost, different assessment intentions lead to such different answers that there is a strong case for regarding feedout and feedback assessment as different systems. When the intention is to create feedback, learners need to be willing participants who disclose their uncertainties, errors and lacunae. Where feedout is the goal, disclosure is displaced by deception. Players in assessment games need to know which rules apply and there is little room to apply both sets to the same task. Not only do summative and formative assessments instantiate different rules of engagement, because they have different intentions, they also have different rules of evidence. Similarities are identified in the following list:
1. Assessors look for evidence of achievement.
2. Judgements are made about the match between evidence and criteria. In many cases conclusions are much less reliable, because the goodness of fit between evidence and criteria has to be inferred.
3. Judgement invokes communication. Some assessment judgements are relatively low stakes, may be fuzzy and exploratory and conversational in character, as in a discussion amongst peers or in dialogue with a tutor. In such cases there is often space on all sides for meanings to be negotiated and clarified. High stakes assessments are marked by unequal power relations between assessor and assessed, the format is not conversational and the judgement may be in a highly symbolic form, such as a number or letter grade, that is capable of multiple decodings. Understood as communications, summative and formative assessment are very different. The former is about conveying information, while the latter is more hermeneutic.
4. Judgements are economic processes. When people are assessing or being assessed, it is at the expense of doing other things. Stress (amongst teachers and learners) and dysfunctional behaviours (of which surface learning strategies can be one) are common consequences. And although formative assessment is intended to enrich the learning economy, summative assessment’s contribution to learning is less palpable.

Copied from
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86.

Feedout function of assessment (summative assessment)

When assessment certifies or warrants achievement it has a feedout function, in that the grades and classifications can then be treated as a performance indicator for the student, department, institution, employer, funding body, quality agency or compilers of league tables. So important are those feedout functions that such assessment is often called high stakes or summative assessment, and great emphasis is consequently put on making it robust. Reliability is a crucial feature of robust assessment. Careless or capricious feedout is unethical and can be challenged.

Assessment can also have a feedback function when it is intended to evoke information to help further learning.

Copied from
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86.

Assessment and learning

'The single, strongest influence on learning is surely the assessment procedures... even the form of an examination question or essay topics set can affect how students study … It is also important to remember that entrenched attitudes which support traditional methods of teaching and assessment are hard to change.' (Entwistle quoted in Knight, 2002, p. 276)

References:
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86

What conditions assessment should meet

There is good evidence that student achievement is related, first and foremost, to engagement (Astin, 1997). Engagement does not simply equate to the amount of involvement in and time on task, important though that is. It extends to learners’ engagement in communities of practice, to their involvement in a variety of networks and to the amount and quality of interchanges with others. Cognitive engagements with others are powerful stimuli for learning (learning as social acts, Vygotsky).
The award of a degree should attest that graduates have largely achieved what the curriculum specifies, with the added implication that what has been achieved is, in some measure, transferable.

Notice that these assessments have at least three conditions to meet if they are to be consistent with the account of learning that has just been sketched.
1. They have to be faithful to the curriculum (charged with developing understandings, skills, self-theories and reflectiveness).
2. They must align with the notion that education is concerned with some degree of abstraction, generalisation or transfer.
3. They should not impede student engagement in communities of practice, but should encourage behaviours associated with good learning.


Copied from
Knight, P. (2002). Summative assessment in higher education: practices in disarray. Studies in Higher Education, 27.3, pp. p275-86.

Innovative assessment methods and benefits/challenges when using them

Presentations and other oral/non-written assessments (group/individual presentation, discussion-debate, audio-video, role-play)

Benefits/motivation to use them:

• offer opportunities to engage students with learning, to develop communication skills and to simulate authentic situations not available with traditional written coursework and examinations. 
• Group presentations facilitating group working skills. 
• Oral assessment methods are likely to foster deep learning and lead to better engagement of students with assessment tasks. 
• A common theme was the notion of a public performance being particularly motivating (as well as scary) for students, and the idea of openness and interactivity in questioning leading to critical understanding. 
• Furthermore, students may be involved in the negotiation and setting of assessment criteria, as well as the marking of the presentations (democratising assessment or changing power relationships). 
• Students with special educational needs and from minority backgrounds are likely to perform better in these assessments than in more traditional written ones.

challenges/hindrances

• there is a call for clear criteria because of the subjectivity and unreliability of oral versus written assessment. 
• Students fear and apprehension before oral assessments are maybe scarier than other types of assessment. 
• The public aspect of oral assessments, such as presentations before peers, as crucial in fostering greater engagement with the task, but also in adding to the stress. 
• These types of assessment are likely to be very time-consuming and thus restricted to relatively small classes. 
• While it is trivially true that oral assessment methods are likely to improve oral communication skills more than written assessments, it does not follow that they will also engender deep learning, as was often claimed in the literature reviewed. It is likely that methods that allow more open dialogue (discussion and vivas) will do this better than more closed unidirectional methods (presentations). (p.35)

Portfolios and other unconventional writing assignments (learning journals, project documentation, diaries, reading logs and reflexive notebooks)


Benefits/motivation to use them:
The impetus to use portfolios, learning journals or diaries was related to principles of lifelong learning, active learning and in particular this form of assessment provided a place reflection on experience. This form of assessment was process driven and considered to foster metacognitive skills, self-directed learning and to
integrate learning.

challenges/hindrances:
clear criteria referenced frameworks are necessary to underpin the process. Some question the marking of portfolios and the inter-rater reliability of markers.


Assessment of groups and collaboration (students may work on projects leading to reports, posters and/or presentations)
Benefits/motivation to use them:

• group assessment is seen as helpful for developing learning skills or to instil in students the notion of a learning culture which is collaborative rather than competitive
• produce graduates who have transferable skills, including the ability to work in teams and collaborate
• Collaboration and group assessments may help students to see new perspectives and provide a fuller consideration of issues, helping them to develop their ability to understand, analyse and interpret. Self- and peerassessment is often seen within this context as helping students to reflect on their experiences of group work. And providing an assessment element may encourage participation in group tasks.

challenges/hindrances:

• Giving grades for group work was considered important if students were to take the task seriously
• how to allocate marks to groups and in what sense marks could be considered 'fair' for any individual student.
• how far group work might benefit weaker students or hinder brighter students, and whether that mattered. 
• The issue of what was being assessed, whether it was the product or the process or both, was considered and whether the skills needed to produce good group assignments were as valid as those that produce good individual work.

Student involvement in assessment (peer-self assessment)
Benefits/motivation to use them:

• empowering learners
• encouraging attention (relating to oral presentations), dialogue/creating
• partnerships, more proactive strategies in teachers and students,
• reflection
• enhancement of learning, performance, personal development/autonomy
• improvement of affect, student understanding of assessment, student understanding of criteria.

challenges/hindrances:

• the need for preparation and training for both students and teachers
• The need to create a supportive atmosphere to encourage students to air their concerns was also identified. 
• Time allocated to the implementation and experience in carrying it out were also seen as important (the implementation is time-consuming)

Use of new technology in assessment (where assessment is mediated through or delineated by the

technological medium)
examples of this category include: multiple choice type assessment and its variants, computer mediated communication in the form of discussion boards and possible collaborative work by students, and the ways that could be incorporated into assessments, the creation of a website as an assessment, submission of assessment using multimedia.

Emerging themes from the use of new technology in assessment:

• opens the dialogue about assessment and learning
• highlights the importance of feedback and the desirability of swift feedback for a positive effect on learning
• emphasises the value of aligning assessment with learning outcomes
• highlights the importance of staff familiarity with the medium and the possible provision of training and support
• highlights the importance of feedback, support and facilitation for students, including the provision of opportunities to practise assessment tasks
• emphasises the resource implications and the often ‘front loaded’ nature of introducing technologically based assessment.
• highlights the use of computer-medicated communication and collaborative work as a resource base for assessment.
• the value of ICT in diagnostic testing
• creative possibilities in assessment such as web page design
• the need to design assessment to minimise risk.

implications:

• alignment of assessment to learning outcomes
• clarity of purpose of assessment
• the importance of clear and transparent assessment and marking criteria
• the need for adequate resource to establish electronic assessment
• acknowledgement that the design and implementation of assessment using the electronic medium is often ‘front loaded’ and has implications for staff workload and for student support as it is implemented for the first time
• the need for training of both staff and students in the use of electronic assessment and feedback and the management of the ensuing dialogue about learning.

Conclusion:

Apart from some of the work on the use of ICT to allow students to present their work in the form of a website, it can be seen that although the use of technology may be innovative in assessment, the type of assessment being used may not be so.

What guidance and feedback is given, by what means, and when (This theme concerns innovation in the provision of guidance from tutors about expectations for assessed work, and in the generation and provision of feedback to students (in the form of comments and marks, grades or criterion specific ratings) on the quality of the work they have submitted. Guidance and feedback are generally seen as central to the important formative function of assessment, which is crucial in helping students to achieve high-quality learning outcomes)

Benefits/motivation to use them:

• widespread evidence of dissatisfaction on the part of many students with provision of feedback and guidance
• teacher concerns that some students at least seemed to show little interest in engaging with or even scrutinising feedback comments.

References:
Hounsell, D., Blair, S., Falchikov, N., Hounsell, J., Huxham, M., Klampfleitner, M. and Thomson, K. (2007). Innovative Assessment Across the Disciplines: An Analytical Review of the Literature. Final Report. York: Higher Education Academy (HEA).

2 prescribed topics

• what seem to be the main motivators (drivers/stimuli) or hindrances for teachers to use ICT in their assessment practice; epistemological beliefs implication in selecting their assessment method
• how epistemological beliefs (beliefs about knowledge and learning) affect ICT teachers' assessment beliefs and methods and what are their perceptions about students' benefits from their assessment methods

defining innovative assessment methods

Defining what is meant by innovative assessment is notoriously tricky since its meaning varies from one subject area or institution to another and over time. The present review followed an inclusive and pragmatic approach: innovative assessment was taken to be that which was novel in the eyes of its begetters or beholders; entailed more than a minor modification; was not self-evidently traditional; and had been documented over the last decade.

(Hounsel et al., 2007, p.4)

Innovation in higher education has generally been taken to mean a planned or deliberate process of introducing change, directed towards (but not necessarily achieving) improvements or solving or alleviating some perceived problem. Such changes may be new to a person, course department, institution or higher education as a whole. An innovation in one situation may be something already established elsewhere, but its importance for this discussion is that initiative takers and participants see it as an innovation in their circumstances.

(Hannan and Silver quoted in Hounsel et al., 2007, p.16)

innovation is an idea, product, process or service that adds value, and is useful or transforms current practice in the context to which it is applied.

(Southwell et al. quoted in Hounsel et al., 2007, p.16)


References:
Hounsell, D., Blair, S., Falchikov, N., Hounsell, J., Huxham, M., Klampfleitner, M. and Thomson, K. (2007). Innovative Assessment Across the Disciplines: An Analytical Review of the Literature. Final Report. York: Higher Education Academy (HEA). http://www.heacademy.ac.uk/ourwork/research/litreviews

something which could be about my rationale

Much real world research is sparked off by wanting to solve a problem, or a concern for change and improvement in something to do with practice, or curiosity based on something in the media, personal values and everyday life.


references:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.47)

What works best, for whom, and under what circumstances?

I guess that the conclusion should rephrase the question from "What will produce the greatest overall change?" to "What works best, for whom, and under what circumstances?". In other words, the focus shifts to seeking an understanding not only of the combination of circumstances where substantial gains have been achieved, but also where the innovation has been totally ineffective or possibly negative in outcome.

references:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.39)

Issues about change

Should I integrate some questions-issues about change? In other words:

1. how (i.e., describe some ways) can the suggestions from my research be implemented?
2. what are the barriers to implementation?
3. how might they overcome?

references:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.7)

Potential hinder mechanisms to assessment

Some thoughts drawn from realism:

1. maybe teachers can't employ their assessment methods because it is not legitimate (i.e., by the ministry of education, by the pedagogical institure, by discipline counsellor etc.). Or maybe they don't have the available resources (i.e., time, especially if the curriculum is very pressing, or it is only an one-hour per week course) to employ other assessment methods (e.g., it's convenient and time efficient to employ only one test per semester and grade students from that test). (accessibility mechanism?)
2. maybe teachers don't have enough information about assessment methods and they just use "traditional" assessment models (information mechanism). 'Even tutors who are interested in pursuing other goals may lack knowledge of the most appropriate strategies to use and sill in carrying out different kinds of teaching roles. Although they have different aspirations, they are limited to reproducing the model of teaching that they experienced as students' (Toohey, 1999, 67)
   
3. maybe teachers view assessment as a burden (e.g. they are over-stressed from parents-students) and subsequently dont recognise its pedagogic value (a needs and expectation mechanism?)
4. some courses are less significant than others, for example there are no final exams for ICT and music in secondary schools, some courses in high schools are optional and the overall grade is not affected by the grade in the optional courses. Does this affect assessment methods? (a needs and expectation mechanism?)
5. assessment is affected by school context: the curriculum is delivered via interaction in the classroom where there are expectations and rules about the behaviour of both teachers and pupils, which may or may not vary from class to class, teacher to teacher, school to school, etc. (p.39). 
6. Mechanisms can run on various levels: micro and macro, group, organizational, etc. (for more details look at p.38-39).

Which mechanisms of the above appear to be the "bankers" (i.e., those agreed to be the ones to concentrate on)? To what grade the above mechanisms affect my topic?


The desirable end-state of my research should be to come up with one or more postulate mechanisms which are capable of explaining the phenomena; that I have good reason to believe in their existence; and that I can specify the contexts in which these mechanisms operate to produce and outcome/effect or in reverse to block the outcome/effect. (p.38)
While we are seeking potential causal explanations which were in operation we should eliminate alternatives which might have been involved (p.40).

references:
Real World Research: A Resource for Social Scientists and Practitioner-Researchers, 2nd Edition. Colin Robson. (p.37-40)

Selecting the topic: first thoughts list

I guess it would be beneficial to jot down some initial thoughts about my research question, so:

• epistemological beliefs, beliefs about intelligence among teachers (who use ICT in their teaching) of secondary-high school education (in Greece) and how these beliefs affect their assessment beliefs and methods (note: they may employ different assessment methods incoherent with their beliefs?)
• how various assessment methods can motivate or demotivate students; assessment affect students' sense of autonomy?
• how students' beliefs affect their preference for assessment methods?
• Should I expand to teachers-students of primary or tertiary education?
• Should I focus only to secondary education or only to high schools?
• Should I focus only to ICT teachers or should I include other disciplines also (provided that they use ICT in their teaching practice)?