Audiovisual discrimination of CV syllables: a simultaneous fMRI-EEG study

We carried out a simultaneous fMRI-EEG experiment based on discriminating syllabic minimal pairs involving three phonological contrasts characterized by different degrees of visual distinctiveness (vocalic labialization, consonantal place of articulation or voicing). Audiovisual CV syllable pairs were presented either with a static facial configuration or with a dynamic display of articulatory movements. In the sound-disturbed MRI environment, the significant improvement of syllabic discrimination achieved in the dynamic audiovisual modality, compared to the static audiovisual modality was associated with activation of the occipito-temporal cortex (MT + V5) bilaterally, and of the left premotor cortex. MT + V5 was activated in response to facial movements independently of their relation to speech, the latter was specifically activated by phonological discrimination. Significant ERP's to syllabic discrimination were recorded around 150 and 250 ms. Our results provide arguments for the involvement of the speech motor cortex in phonological discrimination, and suggest a multimodal representation of speech units.