Multimodal Emotion Recognition using Physiological and Audio-Visual Features

To design more context-aware systems for smart environments, especially smart cities, the psychological user status such as emotion should be considered in addition to environmental information. In this study, we focus on the tourism domain as a typical use case, and propose a multimodal tourist emotion recognition method during the sightseeing. We employ behavioural cues (eye and head/ body movements) and audio-visual features to recognise emotion. As a result of real-world experiments with tourists, we achieved up to 0.71 of average recall score in 3-class emotion recognition task with feature level fusion.