Uncertainty-aware complementary label queries for active learning

Many active learning methods assume that a learner can simply ask for the full annotations of some training data from annotators. These methods mainly try to cut the annotation costs by minimizing the number of annotation actions. Unfortunately, annotating instances exactly in many real-world classification tasks is still expensive. To reduce the cost of a single annotation action, we try to tackle a novel active learning setting, named active learning with complementary labels (ALCL). ALCL learners ask only yes/no questions in some classes. After receiving answers from annotators, ALCL learners obtain a few supervised instances and more training instances with complementary labels, which specify only one of the classes to which the pattern does not belong. There are two challenging issues in ALCL: one is how to sample instances to be queried, and the other is how to learn from these complementary labels and ordinary accurate labels. For the first issue, we propose an uncertainty-based sampling strategy under this novel setup. For the second issue, we upgrade a previous ALCL method to fit our sampling strategy. Experimental results on various datasets demonstrate the superiority of our approaches.