A Bi-directional Multiple Timescales LSTM Model for Grounding of Actions and Verbs

Alexandre Antunes, Alban Laflaquiere, Tetsuya Ogata, Angelo Cangelosi

研究成果: Conference contribution

抄録

In this paper we present a neural architecture to learn a bi-directional mapping between actions and language. We implement a Multiple Timescale Long Short-Term Memory (MT-LSTM) network comprised of 7 layers with different timescale factors, to connect actions to language without explicitly learning an intermediate representation. Instead, the model self-organizes such representations at the level of a slow-varying latent layer, linking action branch and language branch at the center. We train the model in a bi-directional way, learning how to produce a sentence from a certain action sequence input and, simultaneously, how to generate an action sequence given a sentence as input. Furthermore we show this model preserves some of the generalization behaviour of Multiple Timescale Recurrent Neural Networks (MTRNN) in generating sentences and actions that were not explicitly trained. We compare this model with a number of different baseline models, confirming the importance of both the bi-directional training and the multiple timescales architecture. Finally, the network was evaluated on motor actions performed by an iCub robot and their corresponding letter-based description. The results of these experiments are presented at the end of the paper.

本文言語English
ホスト出版物のタイトル2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019
出版社Institute of Electrical and Electronics Engineers Inc.
ページ2614-2621
ページ数8
ISBN(電子版)9781728140049
DOI
出版ステータスPublished - 2019 11
イベント2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019 - Macau, China
継続期間: 2019 11 32019 11 8

出版物シリーズ

名前IEEE International Conference on Intelligent Robots and Systems
ISSN(印刷版)2153-0858
ISSN(電子版)2153-0866

Conference

Conference2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019
CountryChina
CityMacau
Period19/11/319/11/8

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

フィンガープリント 「A Bi-directional Multiple Timescales LSTM Model for Grounding of Actions and Verbs」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル