Learning and association of synaesthesia phenomenon using deep neural networks

Yuki Yamaguchi, Kuniaki Noda, Shun Nishide, Hiroshi G. Okuno, Tetsuya Ogata

研究成果: Conference contribution

抄録

Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.

本文言語English
ホスト出版物のタイトル2013 IEEE/SICE International Symposium on System Integration, SII 2013
出版社IEEE Computer Society
ページ659-664
ページ数6
ISBN(印刷版)9781479926268
DOI
出版ステータスPublished - 2013
イベント2013 6th IEEE/SICE International Symposium on System Integration, SII 2013 - Kobe, Japan
継続期間: 2013 12 152013 12 17

出版物シリーズ

名前2013 IEEE/SICE International Symposium on System Integration, SII 2013

Conference

Conference2013 6th IEEE/SICE International Symposium on System Integration, SII 2013
国/地域Japan
CityKobe
Period13/12/1513/12/17

ASJC Scopus subject areas

  • 制御およびシステム工学

フィンガープリント

「Learning and association of synaesthesia phenomenon using deep neural networks」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル