Adaptive Pruning for Multi-Head Self-Attention

This paper provides an adaptive pruning approach to compress Multi-Head Self Attention (MHSA) models. The main aim is to suppress redundant attention heads, requiring high computational complexity, without substantially affecting performance. Through head pruning, we propose more flexible and efficient models for the object detection tasks. Specifically, we propose to enhance the architectures of the two state-of-the-art MHSA-based models: Bottleneck Transformers (BoTNet) and Attention Augmented Convolutional Networks (AACN). Our approach relies on the alternation between the escalation and the ablation of heads. We selected the less productive heads. We suggest to exploit two and four heads, rather than four and eight heads for BotNet and AACN models, respectively. Our experiments on ImageNet and Pascal VOC datasets prove that our light-weighted architectures are more efficient compared with the original heavy-weighted ones. We reach close performances and achieve faster convergence during training, which allows easier transfer and deployment.