Integrated Learning of Robot Motion and Sentences: Real-Time Prediction of Grasping Motion and Attention based on Language Instructions

Hiroshi Ito, Hideyuki Ichiwara, Kenjiro Yamamoto, Hiroki Mori, Tetsuya Ogata

研究成果: Conference contribution

抄録

We propose a motion generation model that can achieve robust behavior against environmental changes based on language instructions at a low cost. Conventional robots that communicate with humans use a restricted environment and language to build up a mapping between language and motion, and thus need to prepare a huge training set in order to achieve versatility. Our method trains pairs of language, visual, and motor information of the robot, and generates motions in real-time based on the 'attention' of the language instructions. Specifically, the robot generates motions while focusing on the indicated objects by the human when multiple objects are in the field of view. In addition, since position recognition and motion generation of the indicated object are performed in real-time, robust motion generation is possible in response to changes in the object position and lighting conditions. We clarified that features related to the object name and its location are self-organized in the latent (PB: Parametric Bias) space by end-to-end learning of robot motion and sentences. These observations may indicate the importance of integrated learning of robot motion and sentences since such feature representations cannot be obtained by learning motions alone.

本文言語English
ホスト出版物のタイトル2022 IEEE International Conference on Robotics and Automation, ICRA 2022
出版社Institute of Electrical and Electronics Engineers Inc.
ページ5404-5410
ページ数7
ISBN(電子版)9781728196817
DOI
出版ステータスPublished - 2022
イベント39th IEEE International Conference on Robotics and Automation, ICRA 2022 - Philadelphia, United States
継続期間: 2022 5月 232022 5月 27

出版物シリーズ

名前Proceedings - IEEE International Conference on Robotics and Automation
ISSN(印刷版)1050-4729

Conference

Conference39th IEEE International Conference on Robotics and Automation, ICRA 2022
国/地域United States
CityPhiladelphia
Period22/5/2322/5/27

ASJC Scopus subject areas

  • ソフトウェア
  • 制御およびシステム工学
  • 人工知能
  • 電子工学および電気工学

フィンガープリント

「Integrated Learning of Robot Motion and Sentences: Real-Time Prediction of Grasping Motion and Attention based on Language Instructions」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル