Dynamics of the adaptive natural gradient descent method for soft committee machines

Masato Inoue, Hyeyoung Park, Masato Okada

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

The learning efficiency of a simplified version of adaptive natural gradient descent (ANGD) for soft committee machines was evaluated. Statistical-mechanical techniques, which extract order parameters and make the stochastic learning dynamics converge towards deterministic at the large limit of the input dimension N [1,2], were employed. ANGD was found to perform as well as natural gradient descent (NGD). The key condition affecting the learning plateau in ANGD were also revealed.

Original languageEnglish
Article number056120
JournalPhysical Review E - Statistical, Nonlinear, and Soft Matter Physics
Volume69
Issue number5 1
Publication statusPublished - 2004 May
Externally publishedYes

Fingerprint

Gradient Descent Method
Gradient Descent
descent
learning
gradients
Order Parameter
plateaus
Converge
Learning

ASJC Scopus subject areas

  • Physics and Astronomy(all)
  • Condensed Matter Physics
  • Statistical and Nonlinear Physics
  • Mathematical Physics

Cite this

Dynamics of the adaptive natural gradient descent method for soft committee machines. / Inoue, Masato; Park, Hyeyoung; Okada, Masato.

In: Physical Review E - Statistical, Nonlinear, and Soft Matter Physics, Vol. 69, No. 5 1, 056120, 05.2004.

Research output: Contribution to journalArticle

@article{e1273875f12349e097570d90d0b2464e,
title = "Dynamics of the adaptive natural gradient descent method for soft committee machines",
abstract = "The learning efficiency of a simplified version of adaptive natural gradient descent (ANGD) for soft committee machines was evaluated. Statistical-mechanical techniques, which extract order parameters and make the stochastic learning dynamics converge towards deterministic at the large limit of the input dimension N [1,2], were employed. ANGD was found to perform as well as natural gradient descent (NGD). The key condition affecting the learning plateau in ANGD were also revealed.",
author = "Masato Inoue and Hyeyoung Park and Masato Okada",
year = "2004",
month = "5",
language = "English",
volume = "69",
journal = "Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics",
issn = "1063-651X",
publisher = "American Physical Society",
number = "5 1",

}

TY - JOUR

T1 - Dynamics of the adaptive natural gradient descent method for soft committee machines

AU - Inoue, Masato

AU - Park, Hyeyoung

AU - Okada, Masato

PY - 2004/5

Y1 - 2004/5

N2 - The learning efficiency of a simplified version of adaptive natural gradient descent (ANGD) for soft committee machines was evaluated. Statistical-mechanical techniques, which extract order parameters and make the stochastic learning dynamics converge towards deterministic at the large limit of the input dimension N [1,2], were employed. ANGD was found to perform as well as natural gradient descent (NGD). The key condition affecting the learning plateau in ANGD were also revealed.

AB - The learning efficiency of a simplified version of adaptive natural gradient descent (ANGD) for soft committee machines was evaluated. Statistical-mechanical techniques, which extract order parameters and make the stochastic learning dynamics converge towards deterministic at the large limit of the input dimension N [1,2], were employed. ANGD was found to perform as well as natural gradient descent (NGD). The key condition affecting the learning plateau in ANGD were also revealed.

UR - http://www.scopus.com/inward/record.url?scp=42749107566&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=42749107566&partnerID=8YFLogxK

M3 - Article

VL - 69

JO - Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics

JF - Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics

SN - 1063-651X

IS - 5 1

M1 - 056120

ER -