Abstract Summary/Description
There have been concerns regarding the construct validity of reading comprehension tests, particularly when students achieve correct responses to multiple-choice items without reading the accompanying passages. For instance, previous research has indicated correct response rates exceeded chance levels when students did not read the passages (Coleman et al., 2009). However, validating the construct validity of reading comprehension tests through traditional piloting methods requires substantial costs for test administration, data collection, and data analysis. In this study, we aim to address this challenge by utilizing ChatGPT, an advanced generative AI model, responds to multiple-choice reading items without access to the passages. We will request both ChatGPT 3.5 and ChatGPT 4 to generate responses to reading comprehension items without access to the passages and asking for detailed rationales for each response. We hypothesize that given the expansive background knowledge and analytical capabilities of ChatGPT, its correct response rate would represent the upper limit of a typical test taker’s ability to respond to multiple-choice items without accessing the passages. The results will provide insights into the construct validity of the reading comprehension test and the rationales provided by ChatGPT will inform the revision of the test to enhance its construct validity.
