Audiovisual semantic congruency effect with onomatopoeia

Antonio Rei Fidalgo, Kohshe Takahashi, Aiko Murata, Katsumi Watanabe

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    It has been reported that when a congruent natural sound precedes briefly presented visual stimuli, it promotes performance in psychophysics detection tasks. Onomatopoeias refer to words that phonetically mimic or suggest actual sounds. Onomatopoeic words are a form of sound symbolism and are frequently used in Japanese language. In this study, we examined whether the presentation of spoken Japanese onomatopoeia to Japanese native-speakers results in visual detection sensitivity changes. Results indicate that when onomatopoeias are presented 227 ms before a visual stimulus, they have a modulatory audiovisual effect. This effect is closer to the results observed with natural sounds than spoken words, with d' being lower for onomatopoeias when compared with natural sounds. Such suggests that Japanese spoken onomatopoeias may be processed in a manner that is closer to natural sounds than spoken words and points to behavioral consequences of sound symbolism.

    Original languageEnglish
    Title of host publication2018 10th International Conference on Knowledge and Smart Technology
    Subtitle of host publicationCybernetics in the Next Decades, KST 2018
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages254-259
    Number of pages6
    ISBN (Print)9781538640159
    DOIs
    Publication statusPublished - 2018 Aug 6
    Event10th International Conference on Knowledge and Smart Technology, KST 2018 - Chiang Mai, Thailand
    Duration: 2018 Jan 312018 Feb 3

    Other

    Other10th International Conference on Knowledge and Smart Technology, KST 2018
    CountryThailand
    CityChiang Mai
    Period18/1/3118/2/3

    Fingerprint

    Semantics
    Acoustic waves
    Psychophysics
    Congruent
    Sound
    Vision

    Keywords

    • Audiovisual interaction
    • Onomatopoeia
    • Semantic congruency
    • Spoken word
    • Visual detection

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Computer Science Applications
    • Control and Optimization

    Cite this

    Fidalgo, A. R., Takahashi, K., Murata, A., & Watanabe, K. (2018). Audiovisual semantic congruency effect with onomatopoeia. In 2018 10th International Conference on Knowledge and Smart Technology: Cybernetics in the Next Decades, KST 2018 (pp. 254-259). [8426115] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/KST.2018.8426115

    Audiovisual semantic congruency effect with onomatopoeia. / Fidalgo, Antonio Rei; Takahashi, Kohshe; Murata, Aiko; Watanabe, Katsumi.

    2018 10th International Conference on Knowledge and Smart Technology: Cybernetics in the Next Decades, KST 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 254-259 8426115.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Fidalgo, AR, Takahashi, K, Murata, A & Watanabe, K 2018, Audiovisual semantic congruency effect with onomatopoeia. in 2018 10th International Conference on Knowledge and Smart Technology: Cybernetics in the Next Decades, KST 2018., 8426115, Institute of Electrical and Electronics Engineers Inc., pp. 254-259, 10th International Conference on Knowledge and Smart Technology, KST 2018, Chiang Mai, Thailand, 18/1/31. https://doi.org/10.1109/KST.2018.8426115
    Fidalgo AR, Takahashi K, Murata A, Watanabe K. Audiovisual semantic congruency effect with onomatopoeia. In 2018 10th International Conference on Knowledge and Smart Technology: Cybernetics in the Next Decades, KST 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 254-259. 8426115 https://doi.org/10.1109/KST.2018.8426115
    Fidalgo, Antonio Rei ; Takahashi, Kohshe ; Murata, Aiko ; Watanabe, Katsumi. / Audiovisual semantic congruency effect with onomatopoeia. 2018 10th International Conference on Knowledge and Smart Technology: Cybernetics in the Next Decades, KST 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 254-259
    @inproceedings{530f8e6683df4c24a473ab4990faa582,
    title = "Audiovisual semantic congruency effect with onomatopoeia",
    abstract = "It has been reported that when a congruent natural sound precedes briefly presented visual stimuli, it promotes performance in psychophysics detection tasks. Onomatopoeias refer to words that phonetically mimic or suggest actual sounds. Onomatopoeic words are a form of sound symbolism and are frequently used in Japanese language. In this study, we examined whether the presentation of spoken Japanese onomatopoeia to Japanese native-speakers results in visual detection sensitivity changes. Results indicate that when onomatopoeias are presented 227 ms before a visual stimulus, they have a modulatory audiovisual effect. This effect is closer to the results observed with natural sounds than spoken words, with d' being lower for onomatopoeias when compared with natural sounds. Such suggests that Japanese spoken onomatopoeias may be processed in a manner that is closer to natural sounds than spoken words and points to behavioral consequences of sound symbolism.",
    keywords = "Audiovisual interaction, Onomatopoeia, Semantic congruency, Spoken word, Visual detection",
    author = "Fidalgo, {Antonio Rei} and Kohshe Takahashi and Aiko Murata and Katsumi Watanabe",
    year = "2018",
    month = "8",
    day = "6",
    doi = "10.1109/KST.2018.8426115",
    language = "English",
    isbn = "9781538640159",
    pages = "254--259",
    booktitle = "2018 10th International Conference on Knowledge and Smart Technology",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",

    }

    TY - GEN

    T1 - Audiovisual semantic congruency effect with onomatopoeia

    AU - Fidalgo, Antonio Rei

    AU - Takahashi, Kohshe

    AU - Murata, Aiko

    AU - Watanabe, Katsumi

    PY - 2018/8/6

    Y1 - 2018/8/6

    N2 - It has been reported that when a congruent natural sound precedes briefly presented visual stimuli, it promotes performance in psychophysics detection tasks. Onomatopoeias refer to words that phonetically mimic or suggest actual sounds. Onomatopoeic words are a form of sound symbolism and are frequently used in Japanese language. In this study, we examined whether the presentation of spoken Japanese onomatopoeia to Japanese native-speakers results in visual detection sensitivity changes. Results indicate that when onomatopoeias are presented 227 ms before a visual stimulus, they have a modulatory audiovisual effect. This effect is closer to the results observed with natural sounds than spoken words, with d' being lower for onomatopoeias when compared with natural sounds. Such suggests that Japanese spoken onomatopoeias may be processed in a manner that is closer to natural sounds than spoken words and points to behavioral consequences of sound symbolism.

    AB - It has been reported that when a congruent natural sound precedes briefly presented visual stimuli, it promotes performance in psychophysics detection tasks. Onomatopoeias refer to words that phonetically mimic or suggest actual sounds. Onomatopoeic words are a form of sound symbolism and are frequently used in Japanese language. In this study, we examined whether the presentation of spoken Japanese onomatopoeia to Japanese native-speakers results in visual detection sensitivity changes. Results indicate that when onomatopoeias are presented 227 ms before a visual stimulus, they have a modulatory audiovisual effect. This effect is closer to the results observed with natural sounds than spoken words, with d' being lower for onomatopoeias when compared with natural sounds. Such suggests that Japanese spoken onomatopoeias may be processed in a manner that is closer to natural sounds than spoken words and points to behavioral consequences of sound symbolism.

    KW - Audiovisual interaction

    KW - Onomatopoeia

    KW - Semantic congruency

    KW - Spoken word

    KW - Visual detection

    UR - http://www.scopus.com/inward/record.url?scp=85052287411&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85052287411&partnerID=8YFLogxK

    U2 - 10.1109/KST.2018.8426115

    DO - 10.1109/KST.2018.8426115

    M3 - Conference contribution

    AN - SCOPUS:85052287411

    SN - 9781538640159

    SP - 254

    EP - 259

    BT - 2018 10th International Conference on Knowledge and Smart Technology

    PB - Institute of Electrical and Electronics Engineers Inc.

    ER -