Multi-modal Emotion Recognition using Speech Features and Text Embedding

Many studies have been conducted emotion recognition using audio signals as it is easy to collect. However, the accuracy is lower than other methods such as using facial images or video signals. In this paper, we propose an emotion recognition using speech signals and text simultaneously to achieve better performance. For training, we generate 43 feature vectors like mfcc, spectral features and harmonic features from audio data. Also 256 embedding vectors is extracted from text data using pretrained Tacotron encoder. Feature vectors and text embedding vectors are fed into each LSTM layer and frilly connected layer which produces a probability distribution over predicted output classes. By combining the average of both results, the data is assigned to one of four emotion categories : Anger, happiness, sadness, neutrality. Our proposed model outperforms previous state-of-the-art methods when they use Korean emotional speech dataset. © 2021 Korean Institute of Electrical Engineers. All rights reserved.