Online learning dynamics of multilayer perceptrons with unidentifiable parameters

Hyeyoung Park, Masato Inoue, Masato Okada

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

In the over-realizable learning scenario of multilayer perceptions, in which the student network has a larger number of hidden units than the true or optimal network, some of the weight parameters are unidentifiable. In this case, the teacher network consists of a union of optimal subspaces included in the parameter space. The optimal subspaces, which lead to singularities, are known to affect the estimation performance of neural networks. Using statistical mechanics, we investigate the online learning dynamics of two-layer neural networks in the over-realizable scenario with unidentifiable parameters. We show that the convergence speed strongly depends on the initial parameter conditions. We also show that there is a quasi-plateau around the optimal subspace, which differs from the well-known plateaus caused by permutation symmetry. In addition, we discuss the property of the final learning state, relating this to the singular structures.

Original languageEnglish
Pages (from-to)11753-11764
Number of pages12
JournalJournal of Physics A: Mathematical and General
Volume36
Issue number47
DOIs
Publication statusPublished - 2003 Nov 28
Externally publishedYes

    Fingerprint

ASJC Scopus subject areas

  • Physics and Astronomy(all)
  • Statistical and Nonlinear Physics
  • Mathematical Physics

Cite this