Auditory-visual speech perception examined by fMRI and PET

Kaoru Sekiyama, Iwao Kanno, Shuichi Miura, Yoichi Sugita

Research output: Contribution to journalArticle

172 Citations (Scopus)

Abstract

Cross-modal binding in auditory-visual speech perception was investigated by using the McGurk effect, a phenomenon in which hearing is altered by incongruent visual mouth movements. We used functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). In each experiment, the subjects were asked to identify spoken syllables ('ba', 'da', 'ga') presented auditorily, visually, or audiovisually (incongruent stimuli). For the auditory component of the stimuli, there were two conditions of intelligibility (High versus Low) as determined by the signal-to-noise (SN) ratio. The control task was visual talker identification of still faces. In the Low intelligibility condition in which the auditory component of the speech was harder to hear, the visual influence was much stronger. Brain imaging data showed bilateral activations specific to the unimodal auditory stimuli (in the temporal cortex) and visual stimuli (in the MT/V5). For the bimodal audiovisual stimuli, activation in the left temporal cortex extended more posteriorly toward the visual-specific area in the Low intelligibility condition. The direct comparison between the Low and High audiovisual conditions showed increased activations in the posterior part of the left superior temporal sulcus (STS), indicating its relationship with the stronger visual influence. It was discussed that this region is likely to be involved in cross-modal binding of auditory-visual speech.

Original languageEnglish
Pages (from-to)277-287
Number of pages11
JournalNeuroscience Research
Volume47
Issue number3
DOIs
Publication statusPublished - 2003 Nov 1
Externally publishedYes

Keywords

  • Auditory-visual integration
  • Cross-modal binding
  • fMRI
  • PET
  • Speech perception
  • Superior temporal sulcus
  • The McGurk effect

ASJC Scopus subject areas

  • Neuroscience(all)

Fingerprint Dive into the research topics of 'Auditory-visual speech perception examined by fMRI and PET'. Together they form a unique fingerprint.

  • Cite this