Efficient Channel Pruning via Architecture-Guided Search Space Shrinking

Recently, channel pruning methods search for the optimal channel numbers by training a weight-sharing network to evaluate architectures of subnetworks. However, the weight shared between subnetworks incurs severe evaluation bias and an accuracy drop. In this paper, we provide a comprehensive understanding of the search space's impact on the evaluation by dissecting the training process of the weight-sharing network analytically. Specifically, it is proved that the sharing weights induce biased noise on gradients, whose magnitude is proportional to the search range of channel numbers and bias is relative to the average channel numbers of the search space. Motivated by the theoretical result, we design a channel pruning method by training a weight-sharing network with search space shrinking. The search space is iteratively shrunk guided by the optimal architecture searched in the weight-sharing network. The reduced search space boosts the accuracy of the evaluation and significantly cuts down the post-processing computation of finetuning. In the end, we demonstrate the superiority of our channel pruning method over state-of-the-art methods with experiments on ImageNet and COCO.