DECIPHERING STUDENT IDEAS ON THERMODYNAMICS USING COMPUTERIZED LEXICAL ANALYSIS OF STUDENT WRITING

Constructed responses, in which students describe their understanding in their own language, provide better insight into their thinking than do multiple-choice assessments. However, constructed responses are not often employed in large enrollment courses due to the time and resource constraints involved in grading these assessments. In this study, we examined student understanding of thermodynamics using computerized lexical analysis of constructed responses in a large enrollment course (N= 294). Students were asked to interpret a graph depicting changes in free energy during the course of a reaction using both multiple-choice and constructed responses. Constructed responses were analyzed using SPSS Text Analytics for Surveys (TAFS). The software extracts scientific terms from the students' writing and places them into categories using custom dictionaries of science terms. We validated the automated lexical analysis by using the categories created by TAFS as independent variables in discriminant analysis to predict expert scoring of the students' writing. Our findings reveal i) that students hold a heterogeneous mix of correct and incorrect ideas about thermodynamics, and ii) that this heterogeneity is undetected by multiple-choice testing. Almost 50% of the students answering multiple-choice correctly displayed incorrect, or both correct and incorrect conceptualizations in their written responses. Our results support previous studies that have revealed students' heterogeneous ideas about matter and energy conservation and acid-base chemistry using lexical analysis. These findings suggest that computerized lexical analysis can improve instructors' understanding of the heterogeneity of ideas that student harbor about key concepts in STEM disciplines and inform assessment practices