Attention-based multimodal contextual fusion for sentiment and emotion classification using bidirectional LSTM

Due to the availability of an enormous amount of multimodal content on the social web and its applications, automatic sentiment analysis, and emotion detection has become an important and widely researched topic. Improving the quality of multimodal fusion is an important issue in this field of research. In this paper, we present a novel attention-based multimodal contextual fusion strategy, which extract the contextual information among the utterances before fusion. Initially, we fuse two-two modalities at a time and finally, we fuse all three modalities. We use a bidirectional LSTM with an attention model for extracting important contextual information among the utterances. The proposed model was tested on IEMOCAP dataset for emotion classification and CMU-MOSI dataset for sentiment classification. By incorporating the contextual information among utterances in the same video, our proposed method outperforms the existing methods by over 3% in emotion classification and over 2% in sentiment classification.