A CROSS-ATTENTION EMOTION RECOGNITION ALGORITHM BASED ON AUDIO AND VIDEO MODALITIES

In recent years, emotion recognition has received significant attention. In this paper, multimodal information, including speech and facial expressions, is adopted to realize human emotion classification. Firstly, we propose a speech recognition model based on the Parallel convolutional module (Pconv), and an expression emotion recognition model based on the improved Inception-ResnetV2 network. The recognize futures of speech and expression will be further fused by using a cross-attention module coordinated with Bidirectional Long Short-Term Memory (BiLSTM). The experimental results organized on CH-SIMS and CMU-MOSI datasets have demonstrated that the proposed algorithm achieves high recognition accuracy. Each component of this model could contribute to performance improvement in the fair way.