Online learning dynamics of multilayer perceptrons with unidentifiable parameters

Hyeyoung Park, Masato Inoue, Masato Okada

研究成果: Article査読

18 被引用数 (Scopus)

抄録

In the over-realizable learning scenario of multilayer perceptions, in which the student network has a larger number of hidden units than the true or optimal network, some of the weight parameters are unidentifiable. In this case, the teacher network consists of a union of optimal subspaces included in the parameter space. The optimal subspaces, which lead to singularities, are known to affect the estimation performance of neural networks. Using statistical mechanics, we investigate the online learning dynamics of two-layer neural networks in the over-realizable scenario with unidentifiable parameters. We show that the convergence speed strongly depends on the initial parameter conditions. We also show that there is a quasi-plateau around the optimal subspace, which differs from the well-known plateaus caused by permutation symmetry. In addition, we discuss the property of the final learning state, relating this to the singular structures.

本文言語English
ページ(範囲)11753-11764
ページ数12
ジャーナルJournal of Physics A: Mathematical and General
36
47
DOI
出版ステータスPublished - 2003 11 28
外部発表はい

ASJC Scopus subject areas

  • 統計物理学および非線形物理学
  • 数理物理学
  • 物理学および天文学(全般)

フィンガープリント

「Online learning dynamics of multilayer perceptrons with unidentifiable parameters」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル