Abstract Summary/Description
Integrated writing tasks are becoming increasingly prevalent in large-scale tests, especially in higher education admission exams, as they have been shown to mirror authentic academic language use (Weigle & Montee, 2012). Previous studies have identified challenges in assessing integrated writing tasks, such as issues with source use and individual rater effects on rating decisions (Cushing & Tywoniw, 2020; Han et al., 2024). This study investigates raters’ cognitive processes and visual attentiveness when scoring integrated writing tasks from the Georgia State Test of English Proficiency (GSTEP). Using a mixed-methods approach, the research combines eye-tracking data with stimulated recall and follow-up interviews to explore the relationship between raters’ cognitive processes and visual attentiveness during rating. Four trained GSTEP raters were shown ten sets of eight short-answer integrated writing responses from the GSTEP corpus; three responses each addressed a different source text, and two synthesis questions addressed both texts. Raters assigned one content score and one language score per set. Participants’ eye movements were recorded while they viewed responses on a screen. Raters then participated in stimulated recall interviews to reflect on the rating process. Quantitative data consisted of rater scores and eye-tracking metrics, analyzed using descriptive statistics. Qualitative data consisted of stimulated recall and follow-up interviews. Findings revealed that the questions addressing the second source text received the most visual attention, suggesting that the content of source texts and question prompts influence scores. Limitations of this study and implications for future eye-tracking research of integrated writing assessments will be discussed.
