Active Learning for Text Mining from Crowds

The benefits of crowdsourcing have been widely recognized in active learning for text mining. Due to the lack of golden ground-truth, it is crucial to evaluate how trustworthy of "noisy" labelers when labeling informative instances. Despite recent achievements made on active learning with crowdsourcing, most of the research works are involved in tuning a considerable amount of parameters, and also sensitive to noise. In this paper, a novel framework to select both the best-fitting labeler and the most informative instance is proposed, with the help of the minimum description length principle which is acknowledged as noise-tolerant and parameter-free. The algorithm is proved to be effective through extensive experiments on texts.