Looking Back and Ahead: Adaptation and Planning by Gradient Descent

Shingo Murata, Hiroki Sawa, Shigeki Sugano, Tetsuya Ogata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Adaptation and planning are crucial for both biological and artificial agents. In this study, we treat these as an inference problem that we solve using a gradient-based optimization approach. We propose adaptation and planning by gradient descent (APGraDe), a gradient-based computational framework with a hierarchical recurrent neural network (RNN) for adaptation and planning. This framework computes (counterfactual) prediction errors by looking back on past situations based on actual observations and by looking ahead to future situations based on preferred observations (or goal). The internal state of the higher level of the RNN is optimized in the direction of minimizing these errors. The errors for the past contribute to the adaptation while errors for the future contribute to the planning. The proposed APGraDe framework is implemented in a humanoid robot and the robot performs a ball manipulation task with a human experimenter. Experimental results show that given a particular preference, the robot can adapt to unexpected situations while pursuing its own preference through the planning of future actions.

Original languageEnglish
Title of host publication2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019
EditorsAmir Aly, Estela Bicho, Sofiane Boucenna, Bruno Castro da Silva, Mohamed Chetouani, Angel P. del Pobil, Julien Diard, Stephane Doncieux, Tilbe Goksun, Angela Grimminger, Frank Guerin, Yoshinobu Hagiwara, Lorenzo Jamone, Sinan Kalkan, Bruno Lara, Clement Moulin-Frier, Shingo Murata, Takayuki Nagai, Yukie Nagai, Iris Nomikou, Masaki Ogino, Pierre-Yves Oudeyer, Alfredo F. Pereira, Alexandre Pitti, Joanna Raczaszek-Leonardi, Sebastian Risi, Benjamin Rosman, Yulia Sandamirskaya, Malte Schilling, Alessandra Sciutti, Patricia Shaw, Andrea Soltoggio, Michael Spranger, Tadahiro Taniguchi, Serge Thill, Jochen Triesch, Emre Ugur, Anna-Lisa Vollmer
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages151-156
Number of pages6
ISBN (Electronic)9781538681282
DOIs
Publication statusPublished - 2019 Aug
Event9th Joint IEEE International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019 - Oslo, Norway
Duration: 2019 Aug 192019 Aug 22

Publication series

Name2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019

Conference

Conference9th Joint IEEE International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019
CountryNorway
CityOslo
Period19/8/1919/8/22

Fingerprint

Gradient Descent
Planning
Recurrent neural networks
Recurrent Neural Networks
Robots
Robot
Gradient
Humanoid Robot
Prediction Error
Manipulation
Ball
Internal
Optimization
Experimental Results
Framework

Keywords

  • active inference
  • free-energy principle
  • planning as inference
  • prediction error minimization
  • predictive coding
  • recurrent neural network

ASJC Scopus subject areas

  • Artificial Intelligence
  • Human-Computer Interaction
  • Control and Optimization

Cite this

Murata, S., Sawa, H., Sugano, S., & Ogata, T. (2019). Looking Back and Ahead: Adaptation and Planning by Gradient Descent. In A. Aly, E. Bicho, S. Boucenna, B. Castro da Silva, M. Chetouani, A. P. del Pobil, J. Diard, S. Doncieux, T. Goksun, A. Grimminger, F. Guerin, Y. Hagiwara, L. Jamone, S. Kalkan, B. Lara, C. Moulin-Frier, S. Murata, T. Nagai, Y. Nagai, I. Nomikou, M. Ogino, P-Y. Oudeyer, A. F. Pereira, A. Pitti, J. Raczaszek-Leonardi, S. Risi, B. Rosman, Y. Sandamirskaya, M. Schilling, A. Sciutti, P. Shaw, A. Soltoggio, M. Spranger, T. Taniguchi, S. Thill, J. Triesch, E. Ugur, ... A-L. Vollmer (Eds.), 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019 (pp. 151-156). [8850693] (2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DEVLRN.2019.8850693

Looking Back and Ahead : Adaptation and Planning by Gradient Descent. / Murata, Shingo; Sawa, Hiroki; Sugano, Shigeki; Ogata, Tetsuya.

2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019. ed. / Amir Aly; Estela Bicho; Sofiane Boucenna; Bruno Castro da Silva; Mohamed Chetouani; Angel P. del Pobil; Julien Diard; Stephane Doncieux; Tilbe Goksun; Angela Grimminger; Frank Guerin; Yoshinobu Hagiwara; Lorenzo Jamone; Sinan Kalkan; Bruno Lara; Clement Moulin-Frier; Shingo Murata; Takayuki Nagai; Yukie Nagai; Iris Nomikou; Masaki Ogino; Pierre-Yves Oudeyer; Alfredo F. Pereira; Alexandre Pitti; Joanna Raczaszek-Leonardi; Sebastian Risi; Benjamin Rosman; Yulia Sandamirskaya; Malte Schilling; Alessandra Sciutti; Patricia Shaw; Andrea Soltoggio; Michael Spranger; Tadahiro Taniguchi; Serge Thill; Jochen Triesch; Emre Ugur; Anna-Lisa Vollmer. Institute of Electrical and Electronics Engineers Inc., 2019. p. 151-156 8850693 (2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Murata, S, Sawa, H, Sugano, S & Ogata, T 2019, Looking Back and Ahead: Adaptation and Planning by Gradient Descent. in A Aly, E Bicho, S Boucenna, B Castro da Silva, M Chetouani, AP del Pobil, J Diard, S Doncieux, T Goksun, A Grimminger, F Guerin, Y Hagiwara, L Jamone, S Kalkan, B Lara, C Moulin-Frier, S Murata, T Nagai, Y Nagai, I Nomikou, M Ogino, P-Y Oudeyer, AF Pereira, A Pitti, J Raczaszek-Leonardi, S Risi, B Rosman, Y Sandamirskaya, M Schilling, A Sciutti, P Shaw, A Soltoggio, M Spranger, T Taniguchi, S Thill, J Triesch, E Ugur & A-L Vollmer (eds), 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019., 8850693, 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019, Institute of Electrical and Electronics Engineers Inc., pp. 151-156, 9th Joint IEEE International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019, Oslo, Norway, 19/8/19. https://doi.org/10.1109/DEVLRN.2019.8850693
Murata S, Sawa H, Sugano S, Ogata T. Looking Back and Ahead: Adaptation and Planning by Gradient Descent. In Aly A, Bicho E, Boucenna S, Castro da Silva B, Chetouani M, del Pobil AP, Diard J, Doncieux S, Goksun T, Grimminger A, Guerin F, Hagiwara Y, Jamone L, Kalkan S, Lara B, Moulin-Frier C, Murata S, Nagai T, Nagai Y, Nomikou I, Ogino M, Oudeyer P-Y, Pereira AF, Pitti A, Raczaszek-Leonardi J, Risi S, Rosman B, Sandamirskaya Y, Schilling M, Sciutti A, Shaw P, Soltoggio A, Spranger M, Taniguchi T, Thill S, Triesch J, Ugur E, Vollmer A-L, editors, 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 151-156. 8850693. (2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019). https://doi.org/10.1109/DEVLRN.2019.8850693
Murata, Shingo ; Sawa, Hiroki ; Sugano, Shigeki ; Ogata, Tetsuya. / Looking Back and Ahead : Adaptation and Planning by Gradient Descent. 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019. editor / Amir Aly ; Estela Bicho ; Sofiane Boucenna ; Bruno Castro da Silva ; Mohamed Chetouani ; Angel P. del Pobil ; Julien Diard ; Stephane Doncieux ; Tilbe Goksun ; Angela Grimminger ; Frank Guerin ; Yoshinobu Hagiwara ; Lorenzo Jamone ; Sinan Kalkan ; Bruno Lara ; Clement Moulin-Frier ; Shingo Murata ; Takayuki Nagai ; Yukie Nagai ; Iris Nomikou ; Masaki Ogino ; Pierre-Yves Oudeyer ; Alfredo F. Pereira ; Alexandre Pitti ; Joanna Raczaszek-Leonardi ; Sebastian Risi ; Benjamin Rosman ; Yulia Sandamirskaya ; Malte Schilling ; Alessandra Sciutti ; Patricia Shaw ; Andrea Soltoggio ; Michael Spranger ; Tadahiro Taniguchi ; Serge Thill ; Jochen Triesch ; Emre Ugur ; Anna-Lisa Vollmer. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 151-156 (2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019).
@inproceedings{d1b0890d27fa43e48e748285672491c4,
title = "Looking Back and Ahead: Adaptation and Planning by Gradient Descent",
abstract = "Adaptation and planning are crucial for both biological and artificial agents. In this study, we treat these as an inference problem that we solve using a gradient-based optimization approach. We propose adaptation and planning by gradient descent (APGraDe), a gradient-based computational framework with a hierarchical recurrent neural network (RNN) for adaptation and planning. This framework computes (counterfactual) prediction errors by looking back on past situations based on actual observations and by looking ahead to future situations based on preferred observations (or goal). The internal state of the higher level of the RNN is optimized in the direction of minimizing these errors. The errors for the past contribute to the adaptation while errors for the future contribute to the planning. The proposed APGraDe framework is implemented in a humanoid robot and the robot performs a ball manipulation task with a human experimenter. Experimental results show that given a particular preference, the robot can adapt to unexpected situations while pursuing its own preference through the planning of future actions.",
keywords = "active inference, free-energy principle, planning as inference, prediction error minimization, predictive coding, recurrent neural network",
author = "Shingo Murata and Hiroki Sawa and Shigeki Sugano and Tetsuya Ogata",
year = "2019",
month = "8",
doi = "10.1109/DEVLRN.2019.8850693",
language = "English",
series = "2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "151--156",
editor = "Amir Aly and Estela Bicho and Sofiane Boucenna and {Castro da Silva}, Bruno and Mohamed Chetouani and {del Pobil}, {Angel P.} and Julien Diard and Stephane Doncieux and Tilbe Goksun and Angela Grimminger and Frank Guerin and Yoshinobu Hagiwara and Lorenzo Jamone and Sinan Kalkan and Bruno Lara and Clement Moulin-Frier and Shingo Murata and Takayuki Nagai and Yukie Nagai and Iris Nomikou and Masaki Ogino and Pierre-Yves Oudeyer and Pereira, {Alfredo F.} and Alexandre Pitti and Joanna Raczaszek-Leonardi and Sebastian Risi and Benjamin Rosman and Yulia Sandamirskaya and Malte Schilling and Alessandra Sciutti and Patricia Shaw and Andrea Soltoggio and Michael Spranger and Tadahiro Taniguchi and Serge Thill and Jochen Triesch and Emre Ugur and Anna-Lisa Vollmer",
booktitle = "2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019",

}

TY - GEN

T1 - Looking Back and Ahead

T2 - Adaptation and Planning by Gradient Descent

AU - Murata, Shingo

AU - Sawa, Hiroki

AU - Sugano, Shigeki

AU - Ogata, Tetsuya

PY - 2019/8

Y1 - 2019/8

N2 - Adaptation and planning are crucial for both biological and artificial agents. In this study, we treat these as an inference problem that we solve using a gradient-based optimization approach. We propose adaptation and planning by gradient descent (APGraDe), a gradient-based computational framework with a hierarchical recurrent neural network (RNN) for adaptation and planning. This framework computes (counterfactual) prediction errors by looking back on past situations based on actual observations and by looking ahead to future situations based on preferred observations (or goal). The internal state of the higher level of the RNN is optimized in the direction of minimizing these errors. The errors for the past contribute to the adaptation while errors for the future contribute to the planning. The proposed APGraDe framework is implemented in a humanoid robot and the robot performs a ball manipulation task with a human experimenter. Experimental results show that given a particular preference, the robot can adapt to unexpected situations while pursuing its own preference through the planning of future actions.

AB - Adaptation and planning are crucial for both biological and artificial agents. In this study, we treat these as an inference problem that we solve using a gradient-based optimization approach. We propose adaptation and planning by gradient descent (APGraDe), a gradient-based computational framework with a hierarchical recurrent neural network (RNN) for adaptation and planning. This framework computes (counterfactual) prediction errors by looking back on past situations based on actual observations and by looking ahead to future situations based on preferred observations (or goal). The internal state of the higher level of the RNN is optimized in the direction of minimizing these errors. The errors for the past contribute to the adaptation while errors for the future contribute to the planning. The proposed APGraDe framework is implemented in a humanoid robot and the robot performs a ball manipulation task with a human experimenter. Experimental results show that given a particular preference, the robot can adapt to unexpected situations while pursuing its own preference through the planning of future actions.

KW - active inference

KW - free-energy principle

KW - planning as inference

KW - prediction error minimization

KW - predictive coding

KW - recurrent neural network

UR - http://www.scopus.com/inward/record.url?scp=85073672024&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073672024&partnerID=8YFLogxK

U2 - 10.1109/DEVLRN.2019.8850693

DO - 10.1109/DEVLRN.2019.8850693

M3 - Conference contribution

AN - SCOPUS:85073672024

T3 - 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019

SP - 151

EP - 156

BT - 2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2019

A2 - Aly, Amir

A2 - Bicho, Estela

A2 - Boucenna, Sofiane

A2 - Castro da Silva, Bruno

A2 - Chetouani, Mohamed

A2 - del Pobil, Angel P.

A2 - Diard, Julien

A2 - Doncieux, Stephane

A2 - Goksun, Tilbe

A2 - Grimminger, Angela

A2 - Guerin, Frank

A2 - Hagiwara, Yoshinobu

A2 - Jamone, Lorenzo

A2 - Kalkan, Sinan

A2 - Lara, Bruno

A2 - Moulin-Frier, Clement

A2 - Murata, Shingo

A2 - Nagai, Takayuki

A2 - Nagai, Yukie

A2 - Nomikou, Iris

A2 - Ogino, Masaki

A2 - Oudeyer, Pierre-Yves

A2 - Pereira, Alfredo F.

A2 - Pitti, Alexandre

A2 - Raczaszek-Leonardi, Joanna

A2 - Risi, Sebastian

A2 - Rosman, Benjamin

A2 - Sandamirskaya, Yulia

A2 - Schilling, Malte

A2 - Sciutti, Alessandra

A2 - Shaw, Patricia

A2 - Soltoggio, Andrea

A2 - Spranger, Michael

A2 - Taniguchi, Tadahiro

A2 - Thill, Serge

A2 - Triesch, Jochen

A2 - Ugur, Emre

A2 - Vollmer, Anna-Lisa

PB - Institute of Electrical and Electronics Engineers Inc.

ER -