Emotion detection using convolutional neural network and long short-term memory: a deep multimodal framework

Emotion detection systems play a crucial role in enhancing human-computer interaction. Existing systems predominantly rely on machine learning techniques. This study introduces a novel emotion detection method that employs deep learning techniques to identify five basic human emotions and the pleasure dimensions (valence) associated with these emotions, using text and keystroke dynamics. To facilitate this, we develop a non-acted dataset, DEKT-345 × 2, which includes text and keystroke features. The dataset is created by inducing emotions in participants under controlled conditions. Deep learning models are subsequently employed to predict a person’s affective state using textual content. Semantic analysis of the text data is achieved by employing the global vector (Glove) representation of words. For both text and keystroke-based analysis, one-dimensional convolutional neural network (Conv1D), long short-term memory (LSTM), sandwich Conv1D, and sandwich LSTM models are employed. The robustness of our proposed method is assessed using the DEKT-345 × 2 dataset, which collects text and keystroke information from 69 participants. Through parameter tuning on training and validation data, we establish models that demonstrate superior performance compared to five related approaches and three machine learning classifiers. Our proposed framework achieves an accuracy of 88.57% using the LSTM model, 80% using the sandwich LSTM model, 71.42% using the Conv1D model, and 51.48% using the sandwich Conv1D model on text data across the five emotion classes.