UNIGE document Scientific Article
previous document  unige:75900  next document
add to browser collection

Prediction across sensory modalities: A neurocomputational model of the McGurk effect

Published in Cortex. 2015, vol. 68, p. 61-75
Abstract The McGurk effect is a textbook illustration of the automaticity with which the human brain integrates audio-visual speech. It shows that even incongruent audiovisual (AV) speech stimuli can be combined into percepts that correspond neither to the auditory nor to the visual input, but to a mix of both. Typically, when presented with, e.g., visual /aga/ and acoustic /aba/ we perceive an illusory /ada/. In the inverse situation, however, when acoustic /aga/ is paired with visual /aba/, we perceive a combination of both stimuli, i.e., /abga/ or /agba/. Here we assessed the role of dynamic cross-modal predictions in the outcome of AV speech integration using a computational model that processes continuous audiovisual speech sensory inputs in a predictive coding framework. The model involves three processing levels: sensory units, units that encode the dynamics of stimuli, and multimodal recognition/identity units. The model exhibits a dynamic prediction behavior because evidence about speech tokens can be asynchronous across sensory modality, allowing for updating the activity of the recognition units from one modality while sending top-down predictions to the other modality. We explored the model's response to congruent and incongruent AV stimuli and found that, in the two-dimensional feature space spanned by the speech second formant and lip aperture, fusion stimuli are located in the neighborhood of congruent /ada/, which therefore provides a valid match. Conversely, stimuli that lead to combination percepts do not have a unique valid neighbor. In that case, acoustic and visual cues are both highly salient and generate conflicting predictions in the other modality that cannot be fused, forcing the elaboration of a combinatorial solution. We propose that dynamic predictive mechanisms play a decisive role in the dichotomous perception of incongruent audiovisual inputs.
PMID: 26009260
Full text
Article (Published version) (2.2 MB) - public document Free access
Research group Groupe Anne-Lise Giraud (939)
(ISO format)
OLASAGASTI RODRIGUEZ, Miren Itsaso, BOUTON, Sophie, GIRAUD MAMESSIER, Anne-Lise. Prediction across sensory modalities: A neurocomputational model of the McGurk effect. In: Cortex, 2015, vol. 68, p. 61-75. https://archive-ouverte.unige.ch/unige:75900

159 hits



Deposited on : 2015-10-09

Export document
Format :
Citation style :