A Multi-step Attention and Multi-level Structure Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis aims to predict sentiment polarity from several modalities, which is an essential task for widespread applications. The core part of this task is to design a suitable fusion schema to integrate the heterogeneous information from different modalities. However, previous methods usually adopted simple interaction strategies, such as gate or attention mechanisms, which may lead to extracted features containing redundant information. In addition, most of them only focus on the interaction information between single modality, ignoring the modality pair's interaction information. In this paper, we propose a Multi-step Attention and Multi-level Structure network (MAMS) to address the above problems. Specifically, the multi-step attention mechanism extracts the critical information multiple times during the fusion process, which can reduce the interference of redundant information. Furthermore, the multi-level structure can capture both single modality's and modality pair's interaction information. Experimental results on two datasets (CMU-MOSI and CMU-MOSEI) demonstrate the superiority and effectiveness of our proposed MAMS model.