Huot talks about machine scoring of assessment as the culmination of pursuing reliability. And he is right on that one. The machine will spit out the same results no matter when the test is taken or by whom it’s taken. One hundred percent reliability. Nice.
He notes that validity and reliability are in tension with one another. Strong reliability comes with low validity and strong validity comes with low reliability. That’s an intriguing notion. I can’t believe that it is uniformly correct. I can see in the case of machine scoring that the reliability is high and the validity is clearly suspect because a test written for a machine to score can only measure what the machine is capable of measuring (range of vocab, number of words, grammatical accuracy [to some extent], number of paragraphs), but it can’t measure the strength of ideas and logic or the relevance to the task.
It raises the question of how you can optimize both reliability and validity and if it is in fact possible. An intriguing notion and one worth pursuing.
Speaking in an American context Huot talks about funding issues for writing programs. He notes that student numbers are growing while funding is falling. At the same time funding in other areas of universities is growing. Writing programs are the poor siblings of the more valued mainstream programs. This is a problem for Language Centres in many universities. They are underfunded. Their staff aren’t considered academic staff and pay scales and working conditions reflect this.
Another issue to come up in Huot’s interview is portfolios. He says ‘portfolios are the only way for [me] to teach writing’ (p. 104). In my view portfolios are a breeding ground for plagiarism, but I take Huot’s point that time-limited writing tests are not tests of writing ability so much as tests of writing ability in a limited time – which may be an unrealistic representation of what a student is capable of. So this is another area that is problematic.
Bowman, M., Mahon, W., Pogell, S. (2004) Assessment as Opportunity: A Conversation with Brian Huot. Issues in Writing. Vol.14 (2) 94-114