Identifying Useful Features to Detect Off-Topic Essays in Automated ScoringWithout Using Topic-Specific Training Essays

E-rater (R) is the automated scoring engine used at ETS to score the writing quality of essays. A pre-screening filtering system is embedded in e-rater to detect and exclude essays that are not suitable to be scored by e-rater. The pre-screening filtering system is composed of a set of advisory flags, each of which marks some unusualness of the essay (e.g. repetition of words and sentences, restatement of the prompt). This study examined the effectiveness of an advisory flag in the filtering system that detected off-topic essays. The detection of off-topic essays usually requires topic specific training essays to train the engine in order to identify essays that are very different from the other essays of the same topic. The advisory flag used here is designed to detect off-topic essays without using topic-specific training essays because topic-specific training essays may not available in real test settings. To enhance the capability of this off-topic advisory flag, we identified a set of essay features that are potentially useful in distinguishing off-topic essays that do not require topic specific training essays. These features include essay length, the number of word types (exclude non-content-bearing words), the number of word tokens, the similarity of an essay to training essays, essay organization, and the variety of sentences in an essay.