Labeling Quality Problem for Large-Scale Image Recognition

Most CNN models trained on the popular ImageNet dataset are created under the assumption that a single label is used per training image. These models realize remarkable performance on the ImageNet benchmark (with top-1 scores over 90%). Despite this, recognition of several categories is not reliable, as models for these categories can be easily attacked by natural adversarial examples. We show that this effect is related to ambiguous, single labels assigned to training and testing data for these categories. The CNN models tend to learn representations based on parts of an image not related to the label/category. We analyze the labeling scheme used to annotate the popular ImageNet benchmark dataset and compare it with two recent annotation schemes - CloudVision and Real labeling schemes, which are both crowd-sourced annotation efforts. We show that these two schemes lead to a very different granularity of annotations; we also argue that new annotations schemes should not rely on the accuracy on current ImageNet benchmarks as the hint for their correctness (at the Real scheme does).