An exploratory study of the construct measured by automated writing scores across task types and test occasions
Khaled Barkaoui & Johanathan Woodworth, York University
|
https://doi.org/10.58379/QCFS2805
|
Volume 12, Issue 1, 2023
|
Abstract: With the growing use of Automated Essay Scoring (AES) to score second language (L2) writing performance, questions are often raised concerning the interpretation of automated scores. Such questions are typically investigated by examining the association between automated scores and human holistic ratings at one point in time. However, this line of research cannot answer questions about whether and how this association varies across tasks and test occasions. This exploratory study addresses this gap by examining the association between automated writing scores and human multiple-trait ratings of essays written by 48 learners of English in response to TOEFL iBT independent and integrated writing tasks on two test occasions, before and after a period of English language study. Each essay (N= 192) was scored by e-rater and rated by a group of human raters on various writing features. The findings indicated that the associations between e-rater scores and human ratings of some writing features varied significantly across task types and tended to be stronger for essays written after, than for essays written before, English language study. We discuss the findings and their implications for future research on automated scoring in L2 writing assessment.
Keywords: automated essay scoring, human ratings, second language writing, task effects.