Slow dynamics due to singularities of hierarchical learning machines

Hyeyoung Park, Masato Inoue, Masato Okada

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.

Original languageEnglish
Pages (from-to)275-279
Number of pages5
JournalProgress of Theoretical Physics Supplement
Volume157
Publication statusPublished - 2005
Externally publishedYes

Fingerprint

machine learning
learning
plateaus
gradients

ASJC Scopus subject areas

  • Physics and Astronomy(all)

Cite this

Slow dynamics due to singularities of hierarchical learning machines. / Park, Hyeyoung; Inoue, Masato; Okada, Masato.

In: Progress of Theoretical Physics Supplement, Vol. 157, 2005, p. 275-279.

Research output: Contribution to journalArticle

@article{d40ebe1f66ad41ccbbdadad740c1495c,
title = "Slow dynamics due to singularities of hierarchical learning machines",
abstract = "Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.",
author = "Hyeyoung Park and Masato Inoue and Masato Okada",
year = "2005",
language = "English",
volume = "157",
pages = "275--279",
journal = "Progress of Theoretical Physics",
issn = "0033-068X",
publisher = "Yukawa Institute for Theoretical Physics",

}

TY - JOUR

T1 - Slow dynamics due to singularities of hierarchical learning machines

AU - Park, Hyeyoung

AU - Inoue, Masato

AU - Okada, Masato

PY - 2005

Y1 - 2005

N2 - Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.

AB - Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.

UR - http://www.scopus.com/inward/record.url?scp=22144437704&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=22144437704&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:22144437704

VL - 157

SP - 275

EP - 279

JO - Progress of Theoretical Physics

JF - Progress of Theoretical Physics

SN - 0033-068X

ER -