Toward Emotion Recognition and Person Identification Using Lip Movement from Wireless Signals: A Preliminary Study

We present a first of its kind pilot study investigating distinct features presented in lip movement captured through WiFi channel state information (CSI) as four research volunteers read several emotion-charged text samples. We pursue the tasks of emotion and identity recognition with these data via feature-level fusion, extracting features from both the time and frequency domains. While the extracted frequency-domain features, i.e., zero-crossing rate and fundamental frequency, are commonly associated with audio and speech recognition related applications, we found statistical features, such as mean, median, skew, and kurtosis, most suitable for capturing salient information in CSI data. Specifically, classifying the emotional states of speakers (i.e., anger, joy, fear, love, surprise, and sadness) and the identity of speakers themselves, we achieved 96.4% and 57.9% accuracy for the identity and emotion recognition tasks, respectively.