The trouble with “The Trouble with Rubrics”

Kohn (2011) argues that rubrics have a negative effect on student writing because they encourage focus on marks rather than on learning. There is truth to this; they do encourage focus on marks. Students become aware of what is required to attain an A, B, C, D or E. However, the flipside is that the criteria upon which the piece of writing will be judged are laid bare. The marking process is demystified.

First year tertiary students are often working in the dark. A task is given to them along with some of the parameters, but experience suggests that students sometimes don’t know how to read and follow task directions effectively. They don’t know what a good piece of work looks like. They don’t know what is in the assessor’s head. Rubrics reiterate the task parameters and outline the features that are valued and rated at the different levels of achievement. So yes, students are likely to focus on marks, but they are also more likely to focus on task requirements.

On the other hand Kohn alludes to a fundamental flaw in rubrics when he says, “high scores on a list of criteria for excellence in essay writing do not mean that the essay is any good because quality is more than the sum of its rubricized parts” (2011, p. 13). He is right that an essential problem with rubrics is that they do not or cannot capture all of the elements that make a worthy piece of writing. Certainly this is a major issue for rubrics.

Perhaps the major drawbacks of Kohn’s article are that his context appears to be one assessor dealing with one essay and his paradigm is one where authentic assessment is not compatible with awarding marks. The reality for many assessors is that marks have to be awarded, there are hundreds of essays to assess and the assessments must be done under time pressure. Perhaps Kohn is not speaking to this audience. Or perhaps he is calling for a major shake up of all assessment practices and a world where marks are not awarded. The trouble is that this is a major paradigm shift and not one that currently has much groundswell. The reality is that rubrics are flawed in terms of adequately describing what actually happens / should happen in terms of awarding marks. However, the benefits of rubrics are clear too.

—————-

Kohn, A. (2006) The Trouble with Rubrics. English Journal. Vol.95(4) 12-15

Advertisements
Posted in Rubrics | Leave a comment

Rubrics are flawed

Boix Mansiala et al (2009) propose a rubric for interdisciplinary writing. In doing this they note that much progress has been made in terms of ‘how’ to assess, but that ‘what’ to assess remains problematic. Their rubric lays out four dimensions (purposefulness, disciplinary grounding, integration and critical awareness) and rates students at four levels (naive, novice, apprentice and master). However, despite being ‘theoretically grounded’ and ’empirically tested’ the rubric seems as subject to problems with reliability and validity as other rubrics. Like other rubrics it is only as good as ‘what’ it tests, how relevant that ‘what’ is to the task and desired outcomes and how reliably it can produce the same grades by different assessors and / or at different times.

The authors had four assessors mediate and assess several times to refine the rubric. This makes it the best product that the four professionals in question could come up with. Four is a small sample to enable claims about reliability. Validity was tested by comparing the results of students doing interdisciplinary studies to those studying a single discipline. The interdisciplinary students fared better suggesting that the rubric was appropriately measuring some of what the rubric authors wanted to measure. This still leaves unanswered the question (perhaps the unanswerable question) of whether the rubric effectively measures the students’ competence and effectively sorts them into various levels of achievement.

Rubrics are an excellent idea in many ways, but even with the best of intentions and with expert design and input, they remain flawed and are unable to lay strong claims to high levels of reliability and validity. To dispense with them would be to toss the baby out with the bathwater. The question becomes how can we best leverage them.

——————–
Boix Mansilla, V., Duraisingh, E.D., Wolfe, C.R.; Haynes, C (2009) Targeted Assessment Rubric: An Empirically Grounded Rubric for Interdisciplinary Writing. The Journal of Higher Education, Vol. 80(3) 334-353

Posted in Rubrics | Tagged | Leave a comment

Hello world!

Hello world! I’m a spanking new blog. I believe I’m going to be all about assessment issues in higher education. Hence my name.

Posted in Uncategorized | 1 Comment