Learning dynamics of neural networks with singularity - Standard gradient vs. natural gradient

Hyeyoung Park*, Masato Inoue, Masato Okada

*この研究の対応する著者

研究成果: Conference article査読

3 被引用数 (Scopus)

抄録

In hierarchical models, such as neural networks, there exist complex singular structures. The singularity is known to affect estimation performances and learning dynamics of the models. Recently, there have been a number of studies on properties of obtained estimators for the models, but there are few studies on the dynamical properties of learning used for obtaining the estimators. Using two-layer neural networks, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence due to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of a singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity.

本文言語English
ページ(範囲)282-291
ページ数10
ジャーナルLecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)
3157
DOI
出版ステータスPublished - 2004 1月 1
外部発表はい
イベント8th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2004: Trends in Artificial Intelligence - Auckland, New Zealand
継続期間: 2004 8月 92004 8月 13

ASJC Scopus subject areas

  • 理論的コンピュータサイエンス
  • コンピュータ サイエンス(全般)

フィンガープリント

「Learning dynamics of neural networks with singularity - Standard gradient vs. natural gradient」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル