Learning and association of synaesthesia phenomenon using deep neural networks

Yuki Yamaguchi, Kuniaki Noda, Shun Nishide, Hiroshi G. Okuno, Tetsuya Ogata

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.

    Original languageEnglish
    Title of host publication2013 IEEE/SICE International Symposium on System Integration, SII 2013
    PublisherIEEE Computer Society
    Pages659-664
    Number of pages6
    ISBN (Print)9781479926268
    Publication statusPublished - 2013
    Event2013 6th IEEE/SICE International Symposium on System Integration, SII 2013 - Kobe
    Duration: 2013 Dec 152013 Dec 17

    Other

    Other2013 6th IEEE/SICE International Symposium on System Integration, SII 2013
    CityKobe
    Period13/12/1513/12/17

    Fingerprint

    Robots
    Image compression
    Experiments
    Deep neural networks
    Acoustic waves
    Processing

    ASJC Scopus subject areas

    • Control and Systems Engineering

    Cite this

    Yamaguchi, Y., Noda, K., Nishide, S., Okuno, H. G., & Ogata, T. (2013). Learning and association of synaesthesia phenomenon using deep neural networks. In 2013 IEEE/SICE International Symposium on System Integration, SII 2013 (pp. 659-664). [6776750] IEEE Computer Society.

    Learning and association of synaesthesia phenomenon using deep neural networks. / Yamaguchi, Yuki; Noda, Kuniaki; Nishide, Shun; Okuno, Hiroshi G.; Ogata, Tetsuya.

    2013 IEEE/SICE International Symposium on System Integration, SII 2013. IEEE Computer Society, 2013. p. 659-664 6776750.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Yamaguchi, Y, Noda, K, Nishide, S, Okuno, HG & Ogata, T 2013, Learning and association of synaesthesia phenomenon using deep neural networks. in 2013 IEEE/SICE International Symposium on System Integration, SII 2013., 6776750, IEEE Computer Society, pp. 659-664, 2013 6th IEEE/SICE International Symposium on System Integration, SII 2013, Kobe, 13/12/15.
    Yamaguchi Y, Noda K, Nishide S, Okuno HG, Ogata T. Learning and association of synaesthesia phenomenon using deep neural networks. In 2013 IEEE/SICE International Symposium on System Integration, SII 2013. IEEE Computer Society. 2013. p. 659-664. 6776750
    Yamaguchi, Yuki ; Noda, Kuniaki ; Nishide, Shun ; Okuno, Hiroshi G. ; Ogata, Tetsuya. / Learning and association of synaesthesia phenomenon using deep neural networks. 2013 IEEE/SICE International Symposium on System Integration, SII 2013. IEEE Computer Society, 2013. pp. 659-664
    @inproceedings{e712a78d4ddd4ad59bcc533aeefd655c,
    title = "Learning and association of synaesthesia phenomenon using deep neural networks",
    abstract = "Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.",
    author = "Yuki Yamaguchi and Kuniaki Noda and Shun Nishide and Okuno, {Hiroshi G.} and Tetsuya Ogata",
    year = "2013",
    language = "English",
    isbn = "9781479926268",
    pages = "659--664",
    booktitle = "2013 IEEE/SICE International Symposium on System Integration, SII 2013",
    publisher = "IEEE Computer Society",

    }

    TY - GEN

    T1 - Learning and association of synaesthesia phenomenon using deep neural networks

    AU - Yamaguchi, Yuki

    AU - Noda, Kuniaki

    AU - Nishide, Shun

    AU - Okuno, Hiroshi G.

    AU - Ogata, Tetsuya

    PY - 2013

    Y1 - 2013

    N2 - Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.

    AB - Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.

    UR - http://www.scopus.com/inward/record.url?scp=84902491868&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84902491868&partnerID=8YFLogxK

    M3 - Conference contribution

    AN - SCOPUS:84902491868

    SN - 9781479926268

    SP - 659

    EP - 664

    BT - 2013 IEEE/SICE International Symposium on System Integration, SII 2013

    PB - IEEE Computer Society

    ER -