Electrophysiology of Auditory-Visual Speech Integration

Virginie van Wassenhove*, Ken W. Grant, David Poeppel

*Corresponding author for this work

Research output: Contribution to conferencePaperpeer-review

4 Scopus citations


Twenty-six native English Speakers identified auditory (A), visual (V), and congruent and incongruent auditory-visual (AV) syllables while undergoing electroencephalography (EEG) in three experiments. In Experiment 1, unimodal (A, V) and bimodal (AV) stimuli were presented in separate blocks. In Experiment 2, the same stimuli were pseudo-randomized in the same blocks, providing a replication of Experiment 1 while testing the effect of participants' expectancy on the AV condition. In Experiment 3, McGurk fusion (audio/pa/dubbed onto visual/ka/, eliciting the percept/ta/) and combination (audio/ka/dubbed onto visual/pa/) stimuli were tested under visual attention [1]. EEG recordings show early effects of visual influence on auditory evoked-related potentials (P1/N1/P2 complex). Specifically, a robust amplitude reduction of the N1/P2 complex was observed (Experiments 1 and 2) that could not be solely accounted for by attentional effects (Experiment 3). The N1/P2 reduction was accompanied by a temporal facilitation (approximating ~20ms) of the P1/N1and N1/P2 transitions in AV conditions. Additionally, incongruent syllables showed a different profile from congruent AV/ta/over a large latency range (~50 to 350ms post-auditory onset), which was influenced by the accuracy of identification of the visual stimuli presented unimodally. Our results suggest that (i) auditory processing is modulated early on by visual speech inputs, in agreement with an early locus of AV speech interaction, (ii) natural precedence of visual kinematics facilitates auditory speech processing in the time domain, and (iii) the degree of temporal gain is a function of the saliency of visual speech inputs.

Original languageEnglish
Number of pages6
StatePublished - 2003
Externally publishedYes
Event2003 International Conference on Audio-Visual Speech Processing, AVSP 2003 - St. Jorioz, France
Duration: 4 Sep 20037 Sep 2003


Conference2003 International Conference on Audio-Visual Speech Processing, AVSP 2003
CitySt. Jorioz


Dive into the research topics of 'Electrophysiology of Auditory-Visual Speech Integration'. Together they form a unique fingerprint.

Cite this