Facial Expression Recognition by Photo-Reflective Sensors Considering Time Series and Head Posture

There is a method to recognize facial expressions of Head-Mounted Display (HMD) wearers by machine learning of reflection intensity information from photo-reflective sensors embedded into the interior of an HMD [1]. This study evaluates whether facial expression recognition accuracy can be improved by using a learning model that considers temporal changes in sensor values. We assessed whether facial expression recognition accuracy could be improved by adding the head posture data acquired from the Inertial Measurement Unit (IMU) in the HMD to the discriminator input and performing time-series learning. The experimental results showed that PRS-based facial expression recognition with time-series data was more accurate than without. The multimodal recognition using the reflection intensity and head posture data was slightly more accurate than the discrimination using only the reflection intensity information. It was especially effective for the learning condition without considering the time-series.