Cognitive validity in listen-to-write summary tasks: A mixed-methods analysis of notetaking data
Rebecca Yeager, University of Illinois Urbana-Champaign, USA
GoMee Park, University of Texas Rio Grande Valley, USA
Junhee Park, University of Iowa, USA
https://doi.org/10.58379/SPQD9576
Volume 15, Issue 1, 2026
Abstract: This study examines the cognitive validity of a listen-to-summarize task on a university placement test. Cognitive validity is underinvestigated in listening assessment, especially for higher-order listening processes (Holzknecht et al., 2017; Rukthong, 2021). However, student notes from listening tests can record evidence of these processes (Yeager et al., 2024; Cushing, 1991). This study employed an exploratory sequential mixed-methods design to investigate the higher-order listening processes elicited by an integrated listen-to-summarize task. First, notes from two administrations of the same test form – one with notetaking scaffolding (n = 25) and one without (n = 25) – were analyzed qualitatively. Second, qualitative codes informed the development of a rubric to quantify higher-order listening processes across administrations, enabling comparison with summary and multiple-choice scores on another test form (n = 118). Qualitative analysis revealed substantial evidence of higher-order discourse construction processes in student notes, including Selecting, Integrating, Monitoring, and Structure-Building (Field, 2013), especially when scaffolding was provided. Linear regression models indicated that Selecting processes significantly predicted test scores for both the integrated task and the multiple-choice task, while Structure-Building processes were significant only for the integrated task. Results support the cognitive validity of the listen-to-summarize task and inform improved test design and score interpretation. 
Keywords: integrated assessment; listening response formats; English for Academic Purposes; test validation; local testing