The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses

Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants' PEI of the CA, and the empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI.