Online learning dynamics of multilayer perceptrons with unidentifiable parameters

Hyeyoung Park, Masato Inoue, Masato Okada

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

In the over-realizable learning scenario of multilayer perceptions, in which the student network has a larger number of hidden units than the true or optimal network, some of the weight parameters are unidentifiable. In this case, the teacher network consists of a union of optimal subspaces included in the parameter space. The optimal subspaces, which lead to singularities, are known to affect the estimation performance of neural networks. Using statistical mechanics, we investigate the online learning dynamics of two-layer neural networks in the over-realizable scenario with unidentifiable parameters. We show that the convergence speed strongly depends on the initial parameter conditions. We also show that there is a quasi-plateau around the optimal subspace, which differs from the well-known plateaus caused by permutation symmetry. In addition, we discuss the property of the final learning state, relating this to the singular structures.

Original languageEnglish
Pages (from-to)11753-11764
Number of pages12
JournalJournal of Physics A: Mathematical and General
Volume36
Issue number47
DOIs
Publication statusPublished - 2003 Nov 28
Externally publishedYes

Fingerprint

self organizing systems
Online Learning
Multilayer neural networks
Perceptron
learning
Multilayer
Neural networks
Statistical mechanics
Subspace
Multilayers
Neural Networks
Students
plateaus
Scenarios
Convergence Speed
unions
Statistical Mechanics
instructors
permutations
Parameter Space

ASJC Scopus subject areas

  • Physics and Astronomy(all)
  • Statistical and Nonlinear Physics
  • Mathematical Physics

Cite this

Online learning dynamics of multilayer perceptrons with unidentifiable parameters. / Park, Hyeyoung; Inoue, Masato; Okada, Masato.

In: Journal of Physics A: Mathematical and General, Vol. 36, No. 47, 28.11.2003, p. 11753-11764.

Research output: Contribution to journalArticle

@article{c94e9126051843058d000f3b85a6ebe9,
title = "Online learning dynamics of multilayer perceptrons with unidentifiable parameters",
abstract = "In the over-realizable learning scenario of multilayer perceptions, in which the student network has a larger number of hidden units than the true or optimal network, some of the weight parameters are unidentifiable. In this case, the teacher network consists of a union of optimal subspaces included in the parameter space. The optimal subspaces, which lead to singularities, are known to affect the estimation performance of neural networks. Using statistical mechanics, we investigate the online learning dynamics of two-layer neural networks in the over-realizable scenario with unidentifiable parameters. We show that the convergence speed strongly depends on the initial parameter conditions. We also show that there is a quasi-plateau around the optimal subspace, which differs from the well-known plateaus caused by permutation symmetry. In addition, we discuss the property of the final learning state, relating this to the singular structures.",
author = "Hyeyoung Park and Masato Inoue and Masato Okada",
year = "2003",
month = "11",
day = "28",
doi = "10.1088/0305-4470/36/47/004",
language = "English",
volume = "36",
pages = "11753--11764",
journal = "Journal of Physics A: Mathematical and Theoretical",
issn = "1751-8113",
publisher = "IOP Publishing Ltd.",
number = "47",

}

TY - JOUR

T1 - Online learning dynamics of multilayer perceptrons with unidentifiable parameters

AU - Park, Hyeyoung

AU - Inoue, Masato

AU - Okada, Masato

PY - 2003/11/28

Y1 - 2003/11/28

N2 - In the over-realizable learning scenario of multilayer perceptions, in which the student network has a larger number of hidden units than the true or optimal network, some of the weight parameters are unidentifiable. In this case, the teacher network consists of a union of optimal subspaces included in the parameter space. The optimal subspaces, which lead to singularities, are known to affect the estimation performance of neural networks. Using statistical mechanics, we investigate the online learning dynamics of two-layer neural networks in the over-realizable scenario with unidentifiable parameters. We show that the convergence speed strongly depends on the initial parameter conditions. We also show that there is a quasi-plateau around the optimal subspace, which differs from the well-known plateaus caused by permutation symmetry. In addition, we discuss the property of the final learning state, relating this to the singular structures.

AB - In the over-realizable learning scenario of multilayer perceptions, in which the student network has a larger number of hidden units than the true or optimal network, some of the weight parameters are unidentifiable. In this case, the teacher network consists of a union of optimal subspaces included in the parameter space. The optimal subspaces, which lead to singularities, are known to affect the estimation performance of neural networks. Using statistical mechanics, we investigate the online learning dynamics of two-layer neural networks in the over-realizable scenario with unidentifiable parameters. We show that the convergence speed strongly depends on the initial parameter conditions. We also show that there is a quasi-plateau around the optimal subspace, which differs from the well-known plateaus caused by permutation symmetry. In addition, we discuss the property of the final learning state, relating this to the singular structures.

UR - http://www.scopus.com/inward/record.url?scp=0345581643&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0345581643&partnerID=8YFLogxK

U2 - 10.1088/0305-4470/36/47/004

DO - 10.1088/0305-4470/36/47/004

M3 - Article

VL - 36

SP - 11753

EP - 11764

JO - Journal of Physics A: Mathematical and Theoretical

JF - Journal of Physics A: Mathematical and Theoretical

SN - 1751-8113

IS - 47

ER -