Emotional McGurk Effect? A Cross-Cultural Investigation on Emotion Expression under Vocal and Facial Conflict

A multi-modal emotion perceptual experiment is conducted cross-culturally to investigate the difference of expressing and perceiving emotions across Chinese and Japanese. Focus is on the cultural effect on the interaction between the combination of vocal and facial expression and perception. In this paper, part of the perceptual results is reported for the AV-conflicting stimuli produced by a Chinese female speaker and perceived by Chinese and Japanese listeners. The results support the assumptions that (i) When listeners decoding the conflicting AV stimuli, they might rely on some modality more than another across different emotions. (ii) Although common psychological factor contributes to the emotional communication, the decoding of conflicting AV information will be affected by culture background, and (iii) the emotional McGurk effect exists, and it may also be related to cultural norms of the encoder/listener.