Auditory-visual speech perception examined by fMRI and PET

Kaoru Sekiyama, Iwao Kanno, Shuichi Miura, Yoichi Sugita

Research output: Contribution to journalArticle

170 Citations (Scopus)

Abstract

Cross-modal binding in auditory-visual speech perception was investigated by using the McGurk effect, a phenomenon in which hearing is altered by incongruent visual mouth movements. We used functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). In each experiment, the subjects were asked to identify spoken syllables ('ba', 'da', 'ga') presented auditorily, visually, or audiovisually (incongruent stimuli). For the auditory component of the stimuli, there were two conditions of intelligibility (High versus Low) as determined by the signal-to-noise (SN) ratio. The control task was visual talker identification of still faces. In the Low intelligibility condition in which the auditory component of the speech was harder to hear, the visual influence was much stronger. Brain imaging data showed bilateral activations specific to the unimodal auditory stimuli (in the temporal cortex) and visual stimuli (in the MT/V5). For the bimodal audiovisual stimuli, activation in the left temporal cortex extended more posteriorly toward the visual-specific area in the Low intelligibility condition. The direct comparison between the Low and High audiovisual conditions showed increased activations in the posterior part of the left superior temporal sulcus (STS), indicating its relationship with the stronger visual influence. It was discussed that this region is likely to be involved in cross-modal binding of auditory-visual speech.

Original languageEnglish
Pages (from-to)277-287
Number of pages11
JournalNeuroscience Research
Volume47
Issue number3
DOIs
Publication statusPublished - 2003 Nov 1
Externally publishedYes

Fingerprint

Speech Perception
Visual Perception
Temporal Lobe
Positron-Emission Tomography
Magnetic Resonance Imaging
Signal-To-Noise Ratio
Neuroimaging
Hearing
Mouth

Keywords

  • Auditory-visual integration
  • Cross-modal binding
  • fMRI
  • PET
  • Speech perception
  • Superior temporal sulcus
  • The McGurk effect

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Auditory-visual speech perception examined by fMRI and PET. / Sekiyama, Kaoru; Kanno, Iwao; Miura, Shuichi; Sugita, Yoichi.

In: Neuroscience Research, Vol. 47, No. 3, 01.11.2003, p. 277-287.

Research output: Contribution to journalArticle

Sekiyama, K, Kanno, I, Miura, S & Sugita, Y 2003, 'Auditory-visual speech perception examined by fMRI and PET', Neuroscience Research, vol. 47, no. 3, pp. 277-287. https://doi.org/10.1016/S0168-0102(03)00214-1
Sekiyama, Kaoru ; Kanno, Iwao ; Miura, Shuichi ; Sugita, Yoichi. / Auditory-visual speech perception examined by fMRI and PET. In: Neuroscience Research. 2003 ; Vol. 47, No. 3. pp. 277-287.
@article{fda193b17d054fba9da68474b2bce697,
title = "Auditory-visual speech perception examined by fMRI and PET",
abstract = "Cross-modal binding in auditory-visual speech perception was investigated by using the McGurk effect, a phenomenon in which hearing is altered by incongruent visual mouth movements. We used functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). In each experiment, the subjects were asked to identify spoken syllables ('ba', 'da', 'ga') presented auditorily, visually, or audiovisually (incongruent stimuli). For the auditory component of the stimuli, there were two conditions of intelligibility (High versus Low) as determined by the signal-to-noise (SN) ratio. The control task was visual talker identification of still faces. In the Low intelligibility condition in which the auditory component of the speech was harder to hear, the visual influence was much stronger. Brain imaging data showed bilateral activations specific to the unimodal auditory stimuli (in the temporal cortex) and visual stimuli (in the MT/V5). For the bimodal audiovisual stimuli, activation in the left temporal cortex extended more posteriorly toward the visual-specific area in the Low intelligibility condition. The direct comparison between the Low and High audiovisual conditions showed increased activations in the posterior part of the left superior temporal sulcus (STS), indicating its relationship with the stronger visual influence. It was discussed that this region is likely to be involved in cross-modal binding of auditory-visual speech.",
keywords = "Auditory-visual integration, Cross-modal binding, fMRI, PET, Speech perception, Superior temporal sulcus, The McGurk effect",
author = "Kaoru Sekiyama and Iwao Kanno and Shuichi Miura and Yoichi Sugita",
year = "2003",
month = "11",
day = "1",
doi = "10.1016/S0168-0102(03)00214-1",
language = "English",
volume = "47",
pages = "277--287",
journal = "Neuroscience Research",
issn = "0168-0102",
publisher = "Elsevier Ireland Ltd",
number = "3",

}

TY - JOUR

T1 - Auditory-visual speech perception examined by fMRI and PET

AU - Sekiyama, Kaoru

AU - Kanno, Iwao

AU - Miura, Shuichi

AU - Sugita, Yoichi

PY - 2003/11/1

Y1 - 2003/11/1

N2 - Cross-modal binding in auditory-visual speech perception was investigated by using the McGurk effect, a phenomenon in which hearing is altered by incongruent visual mouth movements. We used functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). In each experiment, the subjects were asked to identify spoken syllables ('ba', 'da', 'ga') presented auditorily, visually, or audiovisually (incongruent stimuli). For the auditory component of the stimuli, there were two conditions of intelligibility (High versus Low) as determined by the signal-to-noise (SN) ratio. The control task was visual talker identification of still faces. In the Low intelligibility condition in which the auditory component of the speech was harder to hear, the visual influence was much stronger. Brain imaging data showed bilateral activations specific to the unimodal auditory stimuli (in the temporal cortex) and visual stimuli (in the MT/V5). For the bimodal audiovisual stimuli, activation in the left temporal cortex extended more posteriorly toward the visual-specific area in the Low intelligibility condition. The direct comparison between the Low and High audiovisual conditions showed increased activations in the posterior part of the left superior temporal sulcus (STS), indicating its relationship with the stronger visual influence. It was discussed that this region is likely to be involved in cross-modal binding of auditory-visual speech.

AB - Cross-modal binding in auditory-visual speech perception was investigated by using the McGurk effect, a phenomenon in which hearing is altered by incongruent visual mouth movements. We used functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). In each experiment, the subjects were asked to identify spoken syllables ('ba', 'da', 'ga') presented auditorily, visually, or audiovisually (incongruent stimuli). For the auditory component of the stimuli, there were two conditions of intelligibility (High versus Low) as determined by the signal-to-noise (SN) ratio. The control task was visual talker identification of still faces. In the Low intelligibility condition in which the auditory component of the speech was harder to hear, the visual influence was much stronger. Brain imaging data showed bilateral activations specific to the unimodal auditory stimuli (in the temporal cortex) and visual stimuli (in the MT/V5). For the bimodal audiovisual stimuli, activation in the left temporal cortex extended more posteriorly toward the visual-specific area in the Low intelligibility condition. The direct comparison between the Low and High audiovisual conditions showed increased activations in the posterior part of the left superior temporal sulcus (STS), indicating its relationship with the stronger visual influence. It was discussed that this region is likely to be involved in cross-modal binding of auditory-visual speech.

KW - Auditory-visual integration

KW - Cross-modal binding

KW - fMRI

KW - PET

KW - Speech perception

KW - Superior temporal sulcus

KW - The McGurk effect

UR - http://www.scopus.com/inward/record.url?scp=0142042913&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0142042913&partnerID=8YFLogxK

U2 - 10.1016/S0168-0102(03)00214-1

DO - 10.1016/S0168-0102(03)00214-1

M3 - Article

C2 - 14568109

AN - SCOPUS:0142042913

VL - 47

SP - 277

EP - 287

JO - Neuroscience Research

JF - Neuroscience Research

SN - 0168-0102

IS - 3

ER -