Multimodal integration learning of object manipulation behaviors using deep neural networks

Kuniaki Noda, Hiroaki Arie, Yuki Suga, Tetsuya Ogata

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    10 Citations (Scopus)

    Abstract

    This paper presents a novel computational approach for modeling and generating multiple object manipulation behaviors by a humanoid robot. The contribution of this paper is that deep learning methods are applied not only for multimodal sensor fusion but also for sensory-motor coordination. More specifically, a time-delay deep neural network is applied for modeling multiple behavior patterns represented with multi-dimensional visuomotor temporal sequences. By using the efficient training performance of Hessian-free optimization, the proposed mechanism successfully models six different object manipulation behaviors in a single network. The generalization capability of the learning mechanism enables the acquired model to perform the functions of cross-modal memory retrieval and temporal sequence prediction. The experimental results show that the motion patterns for object manipulation behaviors are successfully generated from the corresponding image sequence, and vice versa. Moreover, the temporal sequence prediction enables the robot to interactively switch multiple behaviors in accordance with changes in the displayed objects.

    Original languageEnglish
    Title of host publicationIEEE International Conference on Intelligent Robots and Systems
    Pages1728-1733
    Number of pages6
    DOIs
    Publication statusPublished - 2013
    Event2013 26th IEEE/RSJ International Conference on Intelligent Robots and Systems: New Horizon, IROS 2013 - Tokyo, Japan
    Duration: 2013 Nov 32013 Nov 8

    Other

    Other2013 26th IEEE/RSJ International Conference on Intelligent Robots and Systems: New Horizon, IROS 2013
    CountryJapan
    CityTokyo
    Period13/11/313/11/8

    Fingerprint

    Robots
    Time delay
    Fusion reactions
    Switches
    Data storage equipment
    Sensors
    Deep neural networks
    Deep learning

    ASJC Scopus subject areas

    • Control and Systems Engineering
    • Software
    • Computer Vision and Pattern Recognition
    • Computer Science Applications

    Cite this

    Noda, K., Arie, H., Suga, Y., & Ogata, T. (2013). Multimodal integration learning of object manipulation behaviors using deep neural networks. In IEEE International Conference on Intelligent Robots and Systems (pp. 1728-1733). [6696582] https://doi.org/10.1109/IROS.2013.6696582

    Multimodal integration learning of object manipulation behaviors using deep neural networks. / Noda, Kuniaki; Arie, Hiroaki; Suga, Yuki; Ogata, Tetsuya.

    IEEE International Conference on Intelligent Robots and Systems. 2013. p. 1728-1733 6696582.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Noda, K, Arie, H, Suga, Y & Ogata, T 2013, Multimodal integration learning of object manipulation behaviors using deep neural networks. in IEEE International Conference on Intelligent Robots and Systems., 6696582, pp. 1728-1733, 2013 26th IEEE/RSJ International Conference on Intelligent Robots and Systems: New Horizon, IROS 2013, Tokyo, Japan, 13/11/3. https://doi.org/10.1109/IROS.2013.6696582
    Noda K, Arie H, Suga Y, Ogata T. Multimodal integration learning of object manipulation behaviors using deep neural networks. In IEEE International Conference on Intelligent Robots and Systems. 2013. p. 1728-1733. 6696582 https://doi.org/10.1109/IROS.2013.6696582
    Noda, Kuniaki ; Arie, Hiroaki ; Suga, Yuki ; Ogata, Tetsuya. / Multimodal integration learning of object manipulation behaviors using deep neural networks. IEEE International Conference on Intelligent Robots and Systems. 2013. pp. 1728-1733
    @inproceedings{91a98bca108f44ca9ee7150a641342f1,
    title = "Multimodal integration learning of object manipulation behaviors using deep neural networks",
    abstract = "This paper presents a novel computational approach for modeling and generating multiple object manipulation behaviors by a humanoid robot. The contribution of this paper is that deep learning methods are applied not only for multimodal sensor fusion but also for sensory-motor coordination. More specifically, a time-delay deep neural network is applied for modeling multiple behavior patterns represented with multi-dimensional visuomotor temporal sequences. By using the efficient training performance of Hessian-free optimization, the proposed mechanism successfully models six different object manipulation behaviors in a single network. The generalization capability of the learning mechanism enables the acquired model to perform the functions of cross-modal memory retrieval and temporal sequence prediction. The experimental results show that the motion patterns for object manipulation behaviors are successfully generated from the corresponding image sequence, and vice versa. Moreover, the temporal sequence prediction enables the robot to interactively switch multiple behaviors in accordance with changes in the displayed objects.",
    author = "Kuniaki Noda and Hiroaki Arie and Yuki Suga and Tetsuya Ogata",
    year = "2013",
    doi = "10.1109/IROS.2013.6696582",
    language = "English",
    isbn = "9781467363587",
    pages = "1728--1733",
    booktitle = "IEEE International Conference on Intelligent Robots and Systems",

    }

    TY - GEN

    T1 - Multimodal integration learning of object manipulation behaviors using deep neural networks

    AU - Noda, Kuniaki

    AU - Arie, Hiroaki

    AU - Suga, Yuki

    AU - Ogata, Tetsuya

    PY - 2013

    Y1 - 2013

    N2 - This paper presents a novel computational approach for modeling and generating multiple object manipulation behaviors by a humanoid robot. The contribution of this paper is that deep learning methods are applied not only for multimodal sensor fusion but also for sensory-motor coordination. More specifically, a time-delay deep neural network is applied for modeling multiple behavior patterns represented with multi-dimensional visuomotor temporal sequences. By using the efficient training performance of Hessian-free optimization, the proposed mechanism successfully models six different object manipulation behaviors in a single network. The generalization capability of the learning mechanism enables the acquired model to perform the functions of cross-modal memory retrieval and temporal sequence prediction. The experimental results show that the motion patterns for object manipulation behaviors are successfully generated from the corresponding image sequence, and vice versa. Moreover, the temporal sequence prediction enables the robot to interactively switch multiple behaviors in accordance with changes in the displayed objects.

    AB - This paper presents a novel computational approach for modeling and generating multiple object manipulation behaviors by a humanoid robot. The contribution of this paper is that deep learning methods are applied not only for multimodal sensor fusion but also for sensory-motor coordination. More specifically, a time-delay deep neural network is applied for modeling multiple behavior patterns represented with multi-dimensional visuomotor temporal sequences. By using the efficient training performance of Hessian-free optimization, the proposed mechanism successfully models six different object manipulation behaviors in a single network. The generalization capability of the learning mechanism enables the acquired model to perform the functions of cross-modal memory retrieval and temporal sequence prediction. The experimental results show that the motion patterns for object manipulation behaviors are successfully generated from the corresponding image sequence, and vice versa. Moreover, the temporal sequence prediction enables the robot to interactively switch multiple behaviors in accordance with changes in the displayed objects.

    UR - http://www.scopus.com/inward/record.url?scp=84893733631&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84893733631&partnerID=8YFLogxK

    U2 - 10.1109/IROS.2013.6696582

    DO - 10.1109/IROS.2013.6696582

    M3 - Conference contribution

    AN - SCOPUS:84893733631

    SN - 9781467363587

    SP - 1728

    EP - 1733

    BT - IEEE International Conference on Intelligent Robots and Systems

    ER -