ALTAANZ
  • About ALTAANZ
  • ALTAANZ Committee
    • Current Committee
    • 2021 Committee
    • 2020 ALTAANZ Committee
    • 2018 - 2019 ALTAANZ Committee
    • 2017 ALTAANZ Committee
    • 2016 ALTAANZ Committee
    • 2015 ALTAANZ Committee
    • 2014 ALTAANZ Committee
    • 2013 ALTAANZ Committee
    • 2012 ALTAANZ Committee
    • 2011 ALTAANZ Committee
  • Joining ALTAANZ
  • ALANZ - ALAA - ALTAANZ 2022
  • ALTAANZ Online Research Forum 2021
  • LTRC/ALTAANZ Online Celebratory event 2020
    • About the event
    • Event Programme
    • LTRC Anniversary Symposium
  • Past ALTAANZ Conferences
    • ALANZ / ALAA / ALTAANZ AUCKLAND 2017
    • ALTAANZ Conference Auckland 2016 >
      • Keynote Speakers >
        • Plenary Abstracts
      • Teachers' Day
      • Pre-conference workshops
      • Conference programme
    • ALTAANZ Conference Brisbane 2014
    • ALTAANZ Conference Sydney 2012
  • Past ALTAANZ / LTRC Workshops
    • LTRC / ALTAANZ Workshops July 2014 >
      • Test analysis for teachers
      • Diagnostic assessment in the language classroom
      • Responding to student writing
      • Assessing Pragmatics
      • Introduction to Rasch measurement
      • Introduction to many-facet Rasch measurement
    • LTRC / ALTAANZ workshops September 2015 >
      • A Practical Approach to Questionnaire Construction for Language Assessment Research
      • Integrating self- and peer-assessment into the language classroom
      • Implementing and assessing collaborative writing activities
      • Assessing Vocabulary
      • Revisiting language constructs
  • Studies in Language Assessment (formerly PLTA)
    • Early View Articles
    • Current Issue
    • About SiLA
    • Past Issues >
      • 2022
      • 2021
      • 2020
      • 2019
      • 2018
      • 2017
      • 2016
      • 2015
      • 2014
      • 2013
      • 2012
    • Editorial Board
    • Contributors
  • ALTAANZ Awards
    • ALTAANZ Best Student Paper Award
    • Penny McKay Award
    • PLTA Best Paper Award
  • Sponsorship for Educational Activities
  • Newsletter: Language Assessment Matters
  • Contact us
Rater variability across examinees and rating criteria in paired speaking assessment
Soo Jung Youn, Northern Arizona University
https://doi/10.0000/0000
Volume 7, Issue 1, 2018
Abstract: This study investigates rater variability with regard to examinees’ levels and rating criteria in paired speaking assessment. 12 raters completed rater training and scored 102 examinees’ paired speaking performances using analytical rating criteria that reflect various features of paired speaking performance. The raters were fairly consistent in their overall ratings, but differed in their severity. The bias analyses using many-facet Rasch measurement revealed that a higher level of rater bias interaction was found for the rating criteria compared to those of the examinees’ levels and the pairing type which reflects a level difference between two examinees. In particular, the most challenging rating category Language Use attracted significant bias interactions. However, the raters did not display more frequent bias interactions based on the interaction-specific rating categories, such as Engaging with Interaction and Turn Organization. Furthermore, the raters tended to reverse their severity patterns across the rating categories. In the rater and examinee bias interactions, the raters tended to show more frequent bias toward the low-level examinees. However, no significant rater bias was found based on the pairing type that consisted of high-level and low-level examinees. These findings have implications for rater training in paired speaking assessment.
Keywords: rater variability, paired speaking assessment, bias analysis, FACETS, rating criteria
Click to download Full Text