Toward abstraction from multi-modal data

Empirical studies on multiple time-scale recurrent models

Junpei Zhong, Angelo Cangelosi, Tetsuya Ogata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

The abstraction tasks are challenging for multi-modal sequences as they require a deeper semantic understanding and a novel text generation for the data. Although the recurrent neural networks (RNN) can be used to model the context of the time-sequences, in most cases the long-term dependencies of multi-modal data make the back-propagation through time training of RNN tend to vanish in the time domain. Recently, inspired from Multiple Time-scale Recurrent Neural Network (MTRNN) [1], an extension of Gated Recurrent Unit (GRU), called Multiple Time-scale Gated Recurrent Unit (MTGRU), has been proposed [2] to learn the long-term dependencies in natural language processing. Particularly it is also able to accomplish the abstraction task for paragraphs given that the time constants are well defined. In this paper, we compare the MTRNN and MTGRU in terms of its learning performances as well as their abstraction representation on higher level (with a slower neural activation). This was done by conducting two studies based on a smaller dataset (two-dimension time sequences from non-linear functions) and a relatively large data-set (43-dimension time sequences from iCub manipulation tasks with multi-modal data). We conclude that gated recurrent mechanisms may be necessary for learning long-term dependencies in large dimension multi-modal data-sets (e.g. learning of robot manipulation), even when natural language commands was not involved. But for smaller learning tasks with simple time-sequences, generic version of recurrent models, such as MTRNN, were sufficient to accomplish the abstraction task.

Original languageEnglish
Title of host publication2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3625-3632
Number of pages8
Volume2017-May
ISBN (Electronic)9781509061815
DOIs
Publication statusPublished - 2017 Jun 30
Externally publishedYes
Event2017 International Joint Conference on Neural Networks, IJCNN 2017 - Anchorage, United States
Duration: 2017 May 142017 May 19

Other

Other2017 International Joint Conference on Neural Networks, IJCNN 2017
CountryUnited States
CityAnchorage
Period17/5/1417/5/19

Fingerprint

Recurrent neural networks
Backpropagation
Chemical activation
Semantics
Robots
Processing

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Cite this

Zhong, J., Cangelosi, A., & Ogata, T. (2017). Toward abstraction from multi-modal data: Empirical studies on multiple time-scale recurrent models. In 2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings (Vol. 2017-May, pp. 3625-3632). [7966312] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IJCNN.2017.7966312

Toward abstraction from multi-modal data : Empirical studies on multiple time-scale recurrent models. / Zhong, Junpei; Cangelosi, Angelo; Ogata, Tetsuya.

2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings. Vol. 2017-May Institute of Electrical and Electronics Engineers Inc., 2017. p. 3625-3632 7966312.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhong, J, Cangelosi, A & Ogata, T 2017, Toward abstraction from multi-modal data: Empirical studies on multiple time-scale recurrent models. in 2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings. vol. 2017-May, 7966312, Institute of Electrical and Electronics Engineers Inc., pp. 3625-3632, 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, United States, 17/5/14. https://doi.org/10.1109/IJCNN.2017.7966312
Zhong J, Cangelosi A, Ogata T. Toward abstraction from multi-modal data: Empirical studies on multiple time-scale recurrent models. In 2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings. Vol. 2017-May. Institute of Electrical and Electronics Engineers Inc. 2017. p. 3625-3632. 7966312 https://doi.org/10.1109/IJCNN.2017.7966312
Zhong, Junpei ; Cangelosi, Angelo ; Ogata, Tetsuya. / Toward abstraction from multi-modal data : Empirical studies on multiple time-scale recurrent models. 2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings. Vol. 2017-May Institute of Electrical and Electronics Engineers Inc., 2017. pp. 3625-3632
@inproceedings{1509e921f4c247c9b39a829473704063,
title = "Toward abstraction from multi-modal data: Empirical studies on multiple time-scale recurrent models",
abstract = "The abstraction tasks are challenging for multi-modal sequences as they require a deeper semantic understanding and a novel text generation for the data. Although the recurrent neural networks (RNN) can be used to model the context of the time-sequences, in most cases the long-term dependencies of multi-modal data make the back-propagation through time training of RNN tend to vanish in the time domain. Recently, inspired from Multiple Time-scale Recurrent Neural Network (MTRNN) [1], an extension of Gated Recurrent Unit (GRU), called Multiple Time-scale Gated Recurrent Unit (MTGRU), has been proposed [2] to learn the long-term dependencies in natural language processing. Particularly it is also able to accomplish the abstraction task for paragraphs given that the time constants are well defined. In this paper, we compare the MTRNN and MTGRU in terms of its learning performances as well as their abstraction representation on higher level (with a slower neural activation). This was done by conducting two studies based on a smaller dataset (two-dimension time sequences from non-linear functions) and a relatively large data-set (43-dimension time sequences from iCub manipulation tasks with multi-modal data). We conclude that gated recurrent mechanisms may be necessary for learning long-term dependencies in large dimension multi-modal data-sets (e.g. learning of robot manipulation), even when natural language commands was not involved. But for smaller learning tasks with simple time-sequences, generic version of recurrent models, such as MTRNN, were sufficient to accomplish the abstraction task.",
author = "Junpei Zhong and Angelo Cangelosi and Tetsuya Ogata",
year = "2017",
month = "6",
day = "30",
doi = "10.1109/IJCNN.2017.7966312",
language = "English",
volume = "2017-May",
pages = "3625--3632",
booktitle = "2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Toward abstraction from multi-modal data

T2 - Empirical studies on multiple time-scale recurrent models

AU - Zhong, Junpei

AU - Cangelosi, Angelo

AU - Ogata, Tetsuya

PY - 2017/6/30

Y1 - 2017/6/30

N2 - The abstraction tasks are challenging for multi-modal sequences as they require a deeper semantic understanding and a novel text generation for the data. Although the recurrent neural networks (RNN) can be used to model the context of the time-sequences, in most cases the long-term dependencies of multi-modal data make the back-propagation through time training of RNN tend to vanish in the time domain. Recently, inspired from Multiple Time-scale Recurrent Neural Network (MTRNN) [1], an extension of Gated Recurrent Unit (GRU), called Multiple Time-scale Gated Recurrent Unit (MTGRU), has been proposed [2] to learn the long-term dependencies in natural language processing. Particularly it is also able to accomplish the abstraction task for paragraphs given that the time constants are well defined. In this paper, we compare the MTRNN and MTGRU in terms of its learning performances as well as their abstraction representation on higher level (with a slower neural activation). This was done by conducting two studies based on a smaller dataset (two-dimension time sequences from non-linear functions) and a relatively large data-set (43-dimension time sequences from iCub manipulation tasks with multi-modal data). We conclude that gated recurrent mechanisms may be necessary for learning long-term dependencies in large dimension multi-modal data-sets (e.g. learning of robot manipulation), even when natural language commands was not involved. But for smaller learning tasks with simple time-sequences, generic version of recurrent models, such as MTRNN, were sufficient to accomplish the abstraction task.

AB - The abstraction tasks are challenging for multi-modal sequences as they require a deeper semantic understanding and a novel text generation for the data. Although the recurrent neural networks (RNN) can be used to model the context of the time-sequences, in most cases the long-term dependencies of multi-modal data make the back-propagation through time training of RNN tend to vanish in the time domain. Recently, inspired from Multiple Time-scale Recurrent Neural Network (MTRNN) [1], an extension of Gated Recurrent Unit (GRU), called Multiple Time-scale Gated Recurrent Unit (MTGRU), has been proposed [2] to learn the long-term dependencies in natural language processing. Particularly it is also able to accomplish the abstraction task for paragraphs given that the time constants are well defined. In this paper, we compare the MTRNN and MTGRU in terms of its learning performances as well as their abstraction representation on higher level (with a slower neural activation). This was done by conducting two studies based on a smaller dataset (two-dimension time sequences from non-linear functions) and a relatively large data-set (43-dimension time sequences from iCub manipulation tasks with multi-modal data). We conclude that gated recurrent mechanisms may be necessary for learning long-term dependencies in large dimension multi-modal data-sets (e.g. learning of robot manipulation), even when natural language commands was not involved. But for smaller learning tasks with simple time-sequences, generic version of recurrent models, such as MTRNN, were sufficient to accomplish the abstraction task.

UR - http://www.scopus.com/inward/record.url?scp=85030972983&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85030972983&partnerID=8YFLogxK

U2 - 10.1109/IJCNN.2017.7966312

DO - 10.1109/IJCNN.2017.7966312

M3 - Conference contribution

VL - 2017-May

SP - 3625

EP - 3632

BT - 2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -