Dynamical integration of language and behavior in a recurrent neural network for Human-Robot interaction

Tatsuro Yamada, Shingo Murata, Hiroaki Arie, Tetsuya Ogata

    Research output: Contribution to journalArticle

    11 Citations (Scopus)

    Abstract

    To work cooperatively with humans by using language, robots must not only acquire a mapping between language and their behavior but also autonomously utilize the mapping in appropriate contexts of interactive tasks online. To this end, we propose a novel learning method linking language to robot behavior by means of a recurrent neural network. In this method, the network learns from correct examples of the imposed task that are given not as explicitly separated sets of language and behavior but as sequential data constructed from the actual temporal flow of the task. By doing this, the internal dynamics of the network models both language-behavior relationships and the temporal patterns of interaction. Here, "internal dynamics" refers to the time development of the system defined on the fixed-dimensional space of the internal states of the context layer. Thus, in the execution phase, by constantly representing where in the interaction context it is as its current state, the network autonomously switches between recognition and generation phases without any explicit signs and utilizes the acquired mapping in appropriate contexts. To evaluate our method, we conducted an experiment in which a robot generates appropriate behavior responding to a human's linguistic instruction. After learning, the network actually formed the attractor structure representing both language-behavior relationships and the task's temporal pattern in its internal dynamics. In the dynamics, language-behavior mapping was achieved by the branching structure. Repetition of human's instruction and robot's behavioral response was represented as the cyclic structure, and besides, waiting to a subsequent instruction was represented as the fixed-point attractor. Thanks to this structure, the robot was able to interact online with a human concerning the given task by autonomously switching phases.

    Original languageEnglish
    Article number00005
    JournalFrontiers in Neurorobotics
    Volume10
    Issue numberJUL
    DOIs
    Publication statusPublished - 2016

    Fingerprint

    Human robot interaction
    Recurrent neural networks
    Robots
    Linguistics
    Switches
    Experiments

    Keywords

    • Dynamical system approach
    • Human-robot interaction
    • Language learning
    • Recurrent neural networks
    • Sequence to sequence learning
    • Symbol grounding problem

    ASJC Scopus subject areas

    • Biomedical Engineering
    • Artificial Intelligence

    Cite this

    Dynamical integration of language and behavior in a recurrent neural network for Human-Robot interaction. / Yamada, Tatsuro; Murata, Shingo; Arie, Hiroaki; Ogata, Tetsuya.

    In: Frontiers in Neurorobotics, Vol. 10, No. JUL, 00005, 2016.

    Research output: Contribution to journalArticle

    @article{ad6acb482b554f9789f15758f7aea2bb,
    title = "Dynamical integration of language and behavior in a recurrent neural network for Human-Robot interaction",
    abstract = "To work cooperatively with humans by using language, robots must not only acquire a mapping between language and their behavior but also autonomously utilize the mapping in appropriate contexts of interactive tasks online. To this end, we propose a novel learning method linking language to robot behavior by means of a recurrent neural network. In this method, the network learns from correct examples of the imposed task that are given not as explicitly separated sets of language and behavior but as sequential data constructed from the actual temporal flow of the task. By doing this, the internal dynamics of the network models both language-behavior relationships and the temporal patterns of interaction. Here, {"}internal dynamics{"} refers to the time development of the system defined on the fixed-dimensional space of the internal states of the context layer. Thus, in the execution phase, by constantly representing where in the interaction context it is as its current state, the network autonomously switches between recognition and generation phases without any explicit signs and utilizes the acquired mapping in appropriate contexts. To evaluate our method, we conducted an experiment in which a robot generates appropriate behavior responding to a human's linguistic instruction. After learning, the network actually formed the attractor structure representing both language-behavior relationships and the task's temporal pattern in its internal dynamics. In the dynamics, language-behavior mapping was achieved by the branching structure. Repetition of human's instruction and robot's behavioral response was represented as the cyclic structure, and besides, waiting to a subsequent instruction was represented as the fixed-point attractor. Thanks to this structure, the robot was able to interact online with a human concerning the given task by autonomously switching phases.",
    keywords = "Dynamical system approach, Human-robot interaction, Language learning, Recurrent neural networks, Sequence to sequence learning, Symbol grounding problem",
    author = "Tatsuro Yamada and Shingo Murata and Hiroaki Arie and Tetsuya Ogata",
    year = "2016",
    doi = "10.3389/fnbot.2016.00005",
    language = "English",
    volume = "10",
    journal = "Frontiers in Neurorobotics",
    issn = "1662-5218",
    publisher = "Frontiers Research Foundation",
    number = "JUL",

    }

    TY - JOUR

    T1 - Dynamical integration of language and behavior in a recurrent neural network for Human-Robot interaction

    AU - Yamada, Tatsuro

    AU - Murata, Shingo

    AU - Arie, Hiroaki

    AU - Ogata, Tetsuya

    PY - 2016

    Y1 - 2016

    N2 - To work cooperatively with humans by using language, robots must not only acquire a mapping between language and their behavior but also autonomously utilize the mapping in appropriate contexts of interactive tasks online. To this end, we propose a novel learning method linking language to robot behavior by means of a recurrent neural network. In this method, the network learns from correct examples of the imposed task that are given not as explicitly separated sets of language and behavior but as sequential data constructed from the actual temporal flow of the task. By doing this, the internal dynamics of the network models both language-behavior relationships and the temporal patterns of interaction. Here, "internal dynamics" refers to the time development of the system defined on the fixed-dimensional space of the internal states of the context layer. Thus, in the execution phase, by constantly representing where in the interaction context it is as its current state, the network autonomously switches between recognition and generation phases without any explicit signs and utilizes the acquired mapping in appropriate contexts. To evaluate our method, we conducted an experiment in which a robot generates appropriate behavior responding to a human's linguistic instruction. After learning, the network actually formed the attractor structure representing both language-behavior relationships and the task's temporal pattern in its internal dynamics. In the dynamics, language-behavior mapping was achieved by the branching structure. Repetition of human's instruction and robot's behavioral response was represented as the cyclic structure, and besides, waiting to a subsequent instruction was represented as the fixed-point attractor. Thanks to this structure, the robot was able to interact online with a human concerning the given task by autonomously switching phases.

    AB - To work cooperatively with humans by using language, robots must not only acquire a mapping between language and their behavior but also autonomously utilize the mapping in appropriate contexts of interactive tasks online. To this end, we propose a novel learning method linking language to robot behavior by means of a recurrent neural network. In this method, the network learns from correct examples of the imposed task that are given not as explicitly separated sets of language and behavior but as sequential data constructed from the actual temporal flow of the task. By doing this, the internal dynamics of the network models both language-behavior relationships and the temporal patterns of interaction. Here, "internal dynamics" refers to the time development of the system defined on the fixed-dimensional space of the internal states of the context layer. Thus, in the execution phase, by constantly representing where in the interaction context it is as its current state, the network autonomously switches between recognition and generation phases without any explicit signs and utilizes the acquired mapping in appropriate contexts. To evaluate our method, we conducted an experiment in which a robot generates appropriate behavior responding to a human's linguistic instruction. After learning, the network actually formed the attractor structure representing both language-behavior relationships and the task's temporal pattern in its internal dynamics. In the dynamics, language-behavior mapping was achieved by the branching structure. Repetition of human's instruction and robot's behavioral response was represented as the cyclic structure, and besides, waiting to a subsequent instruction was represented as the fixed-point attractor. Thanks to this structure, the robot was able to interact online with a human concerning the given task by autonomously switching phases.

    KW - Dynamical system approach

    KW - Human-robot interaction

    KW - Language learning

    KW - Recurrent neural networks

    KW - Sequence to sequence learning

    KW - Symbol grounding problem

    UR - http://www.scopus.com/inward/record.url?scp=84994056007&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84994056007&partnerID=8YFLogxK

    U2 - 10.3389/fnbot.2016.00005

    DO - 10.3389/fnbot.2016.00005

    M3 - Article

    VL - 10

    JO - Frontiers in Neurorobotics

    JF - Frontiers in Neurorobotics

    SN - 1662-5218

    IS - JUL

    M1 - 00005

    ER -