Cross-modal analysis between phonation differences and texture images based on sentiment correlations

Win Thuzar Kyaw, Yoshinori Sagisaka

    Research output: Contribution to journalConference article

    1 Citation (Scopus)

    Abstract

    Motivated by the success of speech characteristics representation by color attributes, we analyzed the cross-modal sentiment correlations between voice source characteristics and textural image characteristics. For the analysis, we employed vowel sounds with representative three phonation differences (modal, creaky and breathy) and 36 texture images with 36 semantic attributes (e.g., banded, cracked and scaly) annotated one semantic attribute for each texture. By asking 40 subjects to select the most fitted textures from 36 figures with different textures after listening 30 speech samples with different phonations, we measured the correlations between acoustic parameters showing voice source variations and the parameters of selected textural image differences showing coarseness, contrast, directionality, busyness, complexity and strength. From the texture classifications, voice characteristics can be roughly characterized by textural differences: modal-gauzy, banded and smeared, creaky-porous, crystalline, cracked and scaly, breathy-smeared, freckled and stained. We have also found significant correlations between voice source acoustic parameters and textural parameters. These correlations suggest the possibility of cross-modal mapping between voice source characteristics and textural parameters, which enables visualization of speech information with source variations reflecting human sentiment perception.

    Original languageEnglish
    Pages (from-to)679-683
    Number of pages5
    JournalProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
    Volume2017-August
    DOIs
    Publication statusPublished - 2017 Jan 1
    Event18th Annual Conference of the International Speech Communication Association, INTERSPEECH 2017 - Stockholm, Sweden
    Duration: 2017 Aug 202017 Aug 24

    Keywords

    • Cross-modal sentiment correlation
    • Phonation differences
    • Textural features
    • Voice source features

    ASJC Scopus subject areas

    • Language and Linguistics
    • Human-Computer Interaction
    • Signal Processing
    • Software
    • Modelling and Simulation

    Fingerprint Dive into the research topics of 'Cross-modal analysis between phonation differences and texture images based on sentiment correlations'. Together they form a unique fingerprint.

  • Cite this