Validity Arguments for Automated Essay Scoring of Young Students' Writing Traits

Machines have a long-demonstrated ability to find statistical relationships between qualities of texts and surface-level linguistic indicators of writing. More recently, unlocked by artificial intelligence, the potential of using machines to identify content-related writing trait criteria has been uncovered. This development is significant, especially in formative assessment contexts where feedback is key. Yet the extent to which writing traits can be validly scored by machines remains under-researched, especially in the K-12 context. The present study investigated the validity of machine learning (ML) models designed for students in grades 3-6 to score three writing traits: task fulfillment, organization and coherence, and vocabulary and expression. The study utilized an argument-based approach, focusing on two primary inferences: evaluation and explanation. The evaluation inference investigated human-machine score alignment, the ability for the models to detect off-topic and gibberish responses, and the consistency of human-machine score alignment across grades and language backgrounds. The explanation inference investigated the relevance of features used in the models. Results indicated that human-machine score alignment was sufficient for all writing traits; however, validity concerns were raised regarding the models' performances detecting off-topic and gibberish responses and the consistency across sub-groups. Implications for language assessment professionals and other educators were discussed.