Multimodal cooperative self-attention network for action recognition

Multimodal human behaviour recognition is a research hotspot in computer vision. To fully use both skeleton and depth data, this paper constructs a new multimodal network identification scheme combined with the self-attention mechanism. The system comprises a transformer-based skeleton self-attention subnetwork and a depth self-attention subnetwork based on CNN. In the skeleton self-attention subnetwork, this paper proposes a motion synergy space feature that can integrate the information of each joint point according to the entirety and synergy of human motion and puts forward a quantitative standard for the contribution degree of each joint motion. In this paper, the results from the skeleton self-attention subnetwork and the depth self-attention subnetwork are integrated and they are verified on the NTU RGB+D and UTD-MHAD datasets. The authors have achieved 90% recognition rate on UTD-MHAD dataset, and the CS recognition rate of the authors' method on the NTU RGB+D dataset reaches 90.5% and the recognition rate of CV is 94.7%. Experimental results show that the network structure proposed in this paper achieves a high recognition rate, and its performance is better than most current methods.