Reducing Classification Cost through Strategic Annotation Assignment

The problem of classifying samples for which there is no definite label is a challenging one in which multiple annotators will provide a more certain input for a classifier. Unlike most of active learning scenarios that require identifying which images to be annotated, we explore how many annotations can potentially be used per instance (one annotation per instance is only the initial step) and propose a threshold-based concept of estimated instance difficulty to guide the custom label acquisition strategy. Using a lung nodule image data set, we determined that, by a simple division of cases into easy and hard to classify, the number of annotations can be distributed to significantly lower the cost (number of acquired annotations) for building a reliable classifier. We show the entire range of available tradeoffs from a small reduction in annotation cost with no perceptible accuracy loss to a large reduction in annotation cost with a minimal sacrifice of classification accuracy.