Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition

Multimodal emotion recognition based on electroencephalogram (EEG) and compensating physiological signals (e.g., eye tracking) has shown potential in the diagnosis and rehabilitation tracking of depression. Since the multi-channel EEG signals are generally processed as one-dimensional (1-D) graph-like features, existing approaches can only adopt underdeveloped shallow models to recognize emotions. However, these simple models have difficulty decoupling complex emotion patterns due to their limited representation capacity. To address this problem, we propose the graph-to-grid (G2G), a concise and plug-and-play module that transforms the 1-D graph-like data into the two-dimensional (2-D) grid-like data via the numerical relation coding. After that, the well developed deep models, e.g., ResNet can be used to downstream tasks. In addition, G2G simplifies the previous complex multimodal fusion into an input matrix augmentation operation, which greatly reduces the difficulty of model design and parameter tuning. Extensive results on three public datasets (SEED, SEED5 and MPED) indicate that the proposed approach achieves state-of-the-art emotion recognition accuracy in both unimodal and multimodal settings, with good cross-session generalization ability. G2G enables the development of more appropriate multimodal emotion recognition algorithms for follow-up studies. Our code is publicly available at https://github.com/Jinminbox/G2G.