Abstract
Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.
| Original language | English |
|---|---|
| Pages (from-to) | 275-279 |
| Number of pages | 5 |
| Journal | Progress of Theoretical Physics Supplement |
| Volume | 157 |
| Publication status | Published - 2005 |
| Externally published | Yes |
Fingerprint
ASJC Scopus subject areas
- Physics and Astronomy(all)
Cite this
Slow dynamics due to singularities of hierarchical learning machines. / Park, Hyeyoung; Inoue, Masato; Okada, Masato.
In: Progress of Theoretical Physics Supplement, Vol. 157, 2005, p. 275-279.Research output: Contribution to journal › Article
}
TY - JOUR
T1 - Slow dynamics due to singularities of hierarchical learning machines
AU - Park, Hyeyoung
AU - Inoue, Masato
AU - Okada, Masato
PY - 2005
Y1 - 2005
N2 - Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.
AB - Recently, slow dynamics in learning of neural networks has been known to be closely related to singularities, which exist in parameter spaces of hierarchical learning models. To show the influence of singular structure on learning dynamics, we take statistical mechanical approaches and investigate online-learning dynamics under various learning scenario with different relationship between optimum and singularities. From the investigation, we found a quasi-plateau phenomenon which differs from the well known plateau. The quasi-plateau and plateau become extremely serious when an optimal point is in a neighborhood of a singularity. The quasi-plateau and plateau disappear in the natural gradient learning, which takes singular structures into account and uses Riemannian measure for the parameter space.
UR - http://www.scopus.com/inward/record.url?scp=22144437704&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=22144437704&partnerID=8YFLogxK
M3 - Article
AN - SCOPUS:22144437704
VL - 157
SP - 275
EP - 279
JO - Progress of Theoretical Physics
JF - Progress of Theoretical Physics
SN - 0033-068X
ER -