In hierarchical models, such as neural networks, there exist complex singular structures. The singularity is known to affect estimation performances and learning dynamics of the models. Recently, there have been a number of studies on properties of obtained estimators for the models, but there are few studies on the dynamical properties of learning used for obtaining the estimators. Using two-layer neural networks, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence due to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of a singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity.
|ジャーナル||Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)|
|出版物ステータス||Published - 2004 12 1|
|イベント||8th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2004: Trends in Artificial Intelligence - Auckland, New Zealand|
継続期間: 2004 8 9 → 2004 8 13
ASJC Scopus subject areas
- Theoretical Computer Science
- Computer Science(all)