Learning dynamics of neural networks with singularity - Standard gradient vs. natural gradient

Hyeyoung Park, Masato Inoue, Masato Okada

研究成果: Conference article

2 引用 (Scopus)


In hierarchical models, such as neural networks, there exist complex singular structures. The singularity is known to affect estimation performances and learning dynamics of the models. Recently, there have been a number of studies on properties of obtained estimators for the models, but there are few studies on the dynamical properties of learning used for obtaining the estimators. Using two-layer neural networks, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence due to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of a singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity.

ジャーナルLecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)
出版物ステータスPublished - 2004 12 1
イベント8th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2004: Trends in Artificial Intelligence - Auckland, New Zealand
継続期間: 2004 8 92004 8 13


ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)