Investigating an online rater training program: product and process
Rosemary Erlam, Janet von Randow & John Read, The University of Auckland
|
https://doi.org/10.58379/AADA5911
|
Volume 2, Issue 1, 2013
|
Abstract: The Diagnostic English Language Needs Assessment (DELNA) programme at the University of Auckland has traditionally offered face-to-face training to both new raters and to experienced raters of writing who need to undergo refresher training in order to increase inter-rater and intra-rater reliability. However, for reasons of practicality and convenience, an online program to offer refresher training for experienced raters was developed and revised after trialling in 2003-04 (Elder, Barkhuizen, Knoch & von Randow, 2007). The study reported on in this paper arose out of the need to establish the validity of using this online DELNA program to train new raters (i.e., novice raters). Rater-training outcomes were investigated through the analysis of quantitative data. However, there was also a focus on the ‘process of rating’ using qualitative data, that is, think-aloud protocol, to identify rating behaviours associated with more vs. less reliable ratings. Results suggest that this program has potential to train novice raters to rate reliably but that future research needs to investigate more rigorously the impact of a program like this one on subsequent rating. Nvivo analysis of the think-aloud protocols highlighted key aspects of the rating behaviours of more and less reliable raters.
Keywords: rater training, rater reliability, writing assessment, online, rater behaviour