Multi-view knowledge distillation for efficient semantic segmentation

Current state-of-the-art semantic segmentation models achieve remarkable success in segmentation accuracy. However, the huge model size and computing cost restrict their applications on low-latency online systems or devices. Knowledge distillation has been one popular solution for compressing large-scale segmentation models, which train a small segmentation model from a large teacher model. However, one teacher model's knowledge may be insufficiently diverse to train an accurate student model. Meanwhile, the student model may inherit bias from the teacher model. This paper proposes a multi-view knowledge distillation framework called MVKD for efficient semantic segmentation. MVKD could aggregate the multi-view knowledge from multiple teacher models and transfer the multi-view knowledge to the student model. In MVKD, we introduce one multi-view co-tuning strategy to acquire uniformity among the multi-view knowledge in features from different teachers. In addition, we propose a multi-view feature distillation loss and a multi-view output distillation loss to transfer the multi-view knowledge in the features and outputs from multiple teachers to the student. We evaluate the proposed MVKD on three benchmark datasets, Cityscapes, CamVid, and Pascal VOC 2012. Experimental results demonstrate the effectiveness of the proposed MVKD in compressing semantic segmentation models.