Machine Scoring – Seriously?

Condon (2011) reviews the various phases of writing assessment over the last century. The developments of the last fifteen or twenty years are particularly interesting. Several important issues are raised including portfolios, machine assessment and the behemoth tests. The relationship between writing theory and assessment theory is discussed as well as the notion of engaging students in the writing process.

PORTFOLIOS
Portfolios first popularized in the 1970s saw a boom in the 1990s and are still popular today. They are favoured as reflecting authentic writing. However, many advocates of portfolios seem to turn a blind eye to a major problem which is the rampant plagiarism that comes with them. Despite all attempts to curb it, plagiarism is a growing issue in tertiary education, and portfolios are a perfect breeding ground.

MACHINE SCORING
Condon expresses dismay at the advent of machine scoring when he briefly mentions COMPASS, E-rater and Criterion. A quick check of their websites reveals that all three state that the essay writing score is received immediately after the test is taken thus confirming the machine marking aspect; http://www.testpreppractice.net/COMPASS/compass-test-format.aspx, http://www.ets.org/erater/about via ScoreitNow!, http://www.ets.org/criterion/higher_ed/about/. Machine scoring can likely claim high levels of reliability (ability to replicate scores at different marking sessions), but questionable levels of validity (whether what is being tested is what should be tested). One can only agree with Condon’s point of view and ask if we can we seriously leave assessment of a communicative skill to a machine.

LARGE SCALE TESTING
The large scale testing business is another controversial issue in assessment and is seen by some as having shortcomings, particularly in terms of compromised validity which it is argued is a result of lack of authenticity partly due to the use of timed essays. In his context Condon mentions a number of American tests which he refers to as ‘assessment juggernauts.’  In an international context, perhaps we can extrapolate the argument to IELTS.

WRITING THEORY & ASSESSMENT THEORY
Condon says, “if writing assessment engages with writing theory, then the assessment practices that emerge will be consistent with the best that has been thought, researched, and written about writing as a construct, as a set of competencies, and as a social practice” (p.173).  This would seem to be a starting point or an enabling factor rather than a clear if/then relationship.

CATERING TO STUDENTS’ INTERESTS
Finally he canvases the argument that the “writing we assess needs to be meaningful to writers” (p. 177). Ideally this would be the case, but it raises questions about what exactly ‘meaningful’ might mean and what the purpose of teaching Academic Writing is. In the world outside the classroom, employees and entrepreneurs prepare documents and written arguments in a range of contexts. They are not necessarily contexts that capture their personal interests or have particular meaning. What they have is context and purpose. The tertiary system should not be treating students with kid-gloves. It should be equipping them with skills for life in the 21st century. This involves research skills, flexibility, initiative, accountability, technological skills, cultural skills and critical thinking, to name a few. Tertiary education needs to engage students, but not necessarily by framing every task so that it caters to them exactly. Students need to take responsibility for their learning too.

It would seem that many of the systems we have in place now represent different philosophical underpinnings, are flawed and may not be capable of being perfected, but they have merits that make them amenable to constructive refinements or even just tinkering with. Others however, such as ‘machine scoring’ seem to be highly ambitious and of questionable value.

======================

Condon, W. (2011) Reinventing Writing Assessment: How the Conversation is Shifting. WPA Vol.34(2) p.162 – 182

Advertisements

About Top English Score

English Test Expert
This entry was posted in writing assessment issues and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s