Learning and association of synaesthesia phenomenon using deep neural networks

Yuki Yamaguchi, Kuniaki Noda, Shun Nishide, Hiroshi G. Okuno, Tetsuya Ogata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.

Original languageEnglish
Title of host publication2013 IEEE/SICE International Symposium on System Integration, SII 2013
PublisherIEEE Computer Society
Pages659-664
Number of pages6
ISBN (Print)9781479926268
Publication statusPublished - 2013 Jan 1
Event2013 6th IEEE/SICE International Symposium on System Integration, SII 2013 - Kobe, Japan
Duration: 2013 Dec 152013 Dec 17

Publication series

Name2013 IEEE/SICE International Symposium on System Integration, SII 2013

Conference

Conference2013 6th IEEE/SICE International Symposium on System Integration, SII 2013
CountryJapan
CityKobe
Period13/12/1513/12/17

ASJC Scopus subject areas

  • Control and Systems Engineering

Fingerprint Dive into the research topics of 'Learning and association of synaesthesia phenomenon using deep neural networks'. Together they form a unique fingerprint.

  • Cite this

    Yamaguchi, Y., Noda, K., Nishide, S., Okuno, H. G., & Ogata, T. (2013). Learning and association of synaesthesia phenomenon using deep neural networks. In 2013 IEEE/SICE International Symposium on System Integration, SII 2013 (pp. 659-664). [6776750] (2013 IEEE/SICE International Symposium on System Integration, SII 2013). IEEE Computer Society.