Semi-automatic coding of open-ended text responses in large-scale assessments

Background In the context of large-scale educational assessments, the effort required to code open-ended text responses is considerably more expensive and time-consuming than the evaluation of multiple-choice responses because it requires trained personnel and long manual coding sessions. Aim Our semi-supervised coding method eco (exploring coding assistant) dynamically supports human raters by automatically coding a subset of the responses. Method We map normalized response texts into a semantic space and cluster response vectors based on their semantic similarity. Assuming that similar codes represent semantically similar responses, we propagate codes to responses in optimally homogeneous clusters. Cluster homogeneity is assessed by strategically querying informative responses and presenting them to a human rater. Following each manual coding, the method estimates the code distribution respecting a certainty interval and assumes a homogeneous distribution if certainty exceeds a predefined threshold. If a cluster is determined to certainly comprise homogeneous responses, all remaining responses are coded accordingly automatically. We evaluated the method in a simulation using different data sets. Results With an average miscoding of about 3%, the method reduced the manual coding effort by an average of about 52%. Conclusion Combining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort.