Audiovisual semantic congruency effect with onomatopoeia

Antonio Rei Fidalgo, Kohshe Takahashi, Aiko Murata, Katsumi Watanabe

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

It has been reported that when a congruent natural sound precedes briefly presented visual stimuli, it promotes performance in psychophysics detection tasks. Onomatopoeias refer to words that phonetically mimic or suggest actual sounds. Onomatopoeic words are a form of sound symbolism and are frequently used in Japanese language. In this study, we examined whether the presentation of spoken Japanese onomatopoeia to Japanese native-speakers results in visual detection sensitivity changes. Results indicate that when onomatopoeias are presented 227 ms before a visual stimulus, they have a modulatory audiovisual effect. This effect is closer to the results observed with natural sounds than spoken words, with d' being lower for onomatopoeias when compared with natural sounds. Such suggests that Japanese spoken onomatopoeias may be processed in a manner that is closer to natural sounds than spoken words and points to behavioral consequences of sound symbolism.

Original languageEnglish
Title of host publication2018 10th International Conference on Knowledge and Smart Technology
Subtitle of host publicationCybernetics in the Next Decades, KST 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages254-259
Number of pages6
ISBN (Print)9781538640159
DOIs
Publication statusPublished - 2018 Aug 6
Event10th International Conference on Knowledge and Smart Technology, KST 2018 - Chiang Mai, Thailand
Duration: 2018 Jan 312018 Feb 3

Publication series

Name2018 10th International Conference on Knowledge and Smart Technology: Cybernetics in the Next Decades, KST 2018

Other

Other10th International Conference on Knowledge and Smart Technology, KST 2018
Country/TerritoryThailand
CityChiang Mai
Period18/1/3118/2/3

Keywords

  • Audiovisual interaction
  • Onomatopoeia
  • Semantic congruency
  • Spoken word
  • Visual detection

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Control and Optimization

Fingerprint

Dive into the research topics of 'Audiovisual semantic congruency effect with onomatopoeia'. Together they form a unique fingerprint.

Cite this