PERCEIVING ASYNCHRONOUS BIMODAL SPEECH IN CONSONANT-VOWEL AND VOWEL SYLLABLES

Subjects naturally integrate auditory and visual information in bimodal speech perception. To assess the robustness of the integration process, the relative onset time of the audible and visible sources was systematically varied. In the first experiment, bimodal syllables composed of the auditory and visible syllables /ba/ and /da/ were present at five different onset asynchronies. The second experiment replicated the same procedure but with the vowels /i/ and /u/. The results indicated that perceivers integrated the two sources of information at all asynchronies. Cluster responses (for example, /bda/ given visual /ba/ and auditory /da/) occurred primarily for the consonants but not for the vowels. In addition, cluster responses require that both the visual and the auditory information be reasonable compatible with the physical properties of a cluster articulation. For both vowels and consonant-vowel syllables, information from the auditory and visual sources is continuous, independent and combined in a three-stage process of feature evaluation, integration and decision.