Positive-Unlabeled Learning for Knowledge Distillation

Convolutional neural networks (CNNs) have greatly promoted the development of artificial intelligence. In general, CNNs with high performance are over-parameterized, requiring massive calculations to process and predict the data. It leads CNNs unable to apply to exiting resource-limited intelligence devices. In this paper, we propose an efficient model compression framework based on knowledge distillation to train a compact student network by a large teacher network. Our key point is to introduce a positive-unlabeled (PU) classifier to promote the compressed student network to learn the features of the prominent teacher network as much as possible. During the training, the PU classifier is to discriminate the features of the teacher network as high-quality and discriminate the features of the student network as low-quality. Simultaneously, the student network learns knowledge from the teacher network through the soft-targets and attention features. Extensive experimental evaluations on four benchmark image classification datasets show that our method outperforms the prior works with a large margin at the same parameters and calculations cost. When selecting the VGGNet19 as the teacher network to train on the CIFAR dataset, the student network VGGNet13 achieves 94.47% and 75.73% accuracy on the CIFAR-10 and CIFAR-100 datasets, which improved 1.02% and 2.44%, respectively.