Semi-automated categorization of open-ended questions

Text data from open-ended questions in surveys are difficult to analyze and are frequently ignored. Yet open-ended questions are important because they do not constrain respondents' answer choices. Where open-ended questions are necessary, sometimes multiple human coders hand-code answers into one of several categories. At the same time, computer scientists have made impressive advances in text mining that may allow automation of such coding. Automated algorithms do not achieve an overall accuracy high enough to entirely replace humans. We categorize easy-to-categorize text answers of open-ended questions automatically using text mining and multinomial boosting, and hard-to-categorize text answers manually. Expected accuracies guide the choice of the threshold delineating between "easy" and "hard" to code text answers. This approach is illustrated with two examples from open-ended questions related to respondents' advice to a patient in a hypothetical dilemma, and a follow-up probe related to respondents' perception of disclosure/privacy risk. Targeting 80% accuracy, we found that 47%-58% of the data could be categorized automatically in research surveys.