Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions

Tatsuro Yamada, Hiroyuki Matsunaga, Tetsuya Ogata

    Research output: Contribution to journalArticle

    8 Citations (Scopus)


    We propose a novel deep learning framework for bidirectional translation between robot actions and their linguistic descriptions. Our model consists of two recurrent autoencoders (RAEs). One RAE learns to encode action sequences as fixed-dimensional vectors in a way that allows the sequences to be reproduced from the vectors by its decoder. The other RAE learns to encode descriptions in a similar way. In the learning process, in addition to reproduction losses, we create another loss function whereby the representations of an action and its corresponding description approach each other in the latent vector space. Across the shared representation, the trained model can produce a linguistic description given a robot action. The model is also able to generate an appropriate action by receiving a linguistic instruction, conditioned on the current visual input. Visualization of the latent representations shows that the robot actions are embedded in a semantically compositional way in the vector space by being learned jointly with descriptions.

    Original languageEnglish
    Article number8403309
    Pages (from-to)3441-3448
    Number of pages8
    JournalIEEE Robotics and Automation Letters
    Issue number4
    Publication statusPublished - 2018 Oct 1



    • AI-based methods
    • Deep learning in robotics and automation
    • neurorobotics

    ASJC Scopus subject areas

    • Control and Systems Engineering
    • Human-Computer Interaction
    • Biomedical Engineering
    • Mechanical Engineering
    • Control and Optimization
    • Artificial Intelligence
    • Computer Science Applications
    • Computer Vision and Pattern Recognition

    Cite this