Using machine learning to model trace behavioral data from a game-based assessment

In the context of game-based assessments (GBAs), we examined the potential of trace data modeling to supplement or replace an existing GBA's scoring approach. We used data science tools and "big data" practices, such as feature engineering and a series of machine learning algorithms, to predict traditionally measured cognitive ability and conscientiousness scores from the copious trace data generated by a theory-driven GBA designed to measure cognitive ability. Several types of predictors were developed from the raw trace data from 621 participants, including counts of game objects that the player interacted with, the amount of time spent doing so, and mouse movement data across a variety of meaningful intervals. Broadly, we found promising evidence for trace data modeling of cognitive ability, including incremental contribution to the prediction of a criterion grade point average (GPA), but less promising evidence for trace data modeling of conscientiousness, suggesting that trace data modeling like this may be more valuable for assessing traits in games that were developed to target those traits.