Epistemic Emotion Detection by Video-based and Heart Rate Variability Features for Online Learning

In this paper, we propose a multimodal method for monitoring epistemic emotions for e-learners. Epistemic emotions are related to knowledge generation and academic performance so it is important to comprehend these emotions to improve teaching and learning efficiency. Epistemic emotions are not always obviously expressed externally so detecting these inner emotions is challenging. In our study, we combine features from videos and heart rate variability (HRV) to train models to estimate six epistemic emotions (Bore, Confuse, Curious, Frustration, Interest and Surprise). The experiment involving 100 Japanese students was conducted remotely, and by adapting the protocol from prior studies, video materials were used as stimuli to trigger epistemic emotions while facial video and HRV were recorded. Our proposed method achieved the average F1-score of 60%, which is superior to using a unimodal method (only video-based or only HRV features). This indicates that the proposed model is a potential solution for monitoring epistemic emotions in e-learning, which is expected to be popular in the future.