A Negative Emotion Recognition System with Internet of Things-Based Multimodal Biosignal Data

Previous studies to recognize negative emotions for mental healthcare have used heavy equipment directly attaching electroencephalogram (EEG) electrodes to the head, and they have proposed binary classification methods to identify negative emotions. To tackle this problem, we propose a negative emotion recognition system to collect multimodal biosignal data such as five EEG signals from an EEG headset and heart rate, galvanic skin response, and skin temperature from a smart band for classifying multiple negative emotions. This consists of an Android Internet of Things (IoT) application, a oneM2M-compliant IoT server, and a machine learning server. The Android IoT application uploads the biosignal data to the IoT server. By using the biosignal data stored in the IoT server, the machine learning server recognizes the negative emotions of disgust, fear, and sadness using a multiclass support vector machine (SVM) model with a radial basis function kernel. The experimental results demonstrate that the multimodal biosignal data approach achieves 93% accuracy. Moreover, when considering only data from the smart band, the system achieved 98% accuracy by optimizing the hyperparameters of the multiclass SVM model. Based on these results, we plan to develop a metaverse system that detects and expresses negative emotions in real time.