A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks

K. R. Müller, M. Finke, Noboru Murata, K. Schulten, S. Amari

Research output: Contribution to journalArticle

23 Citations (Scopus)

Abstract

The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.

Original languageEnglish
Pages (from-to)1085-1106
Number of pages22
JournalNeural Computation
Volume8
Issue number5
Publication statusPublished - 1996 Jul 1
Externally publishedYes

Fingerprint

Learning Curve
Scaling laws
Multilayers
Learning
Backpropagation
Generalization (Psychology)
Scaling

ASJC Scopus subject areas

  • Artificial Intelligence
  • Control and Systems Engineering
  • Neuroscience(all)

Cite this

Müller, K. R., Finke, M., Murata, N., Schulten, K., & Amari, S. (1996). A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks. Neural Computation, 8(5), 1085-1106.

A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks. / Müller, K. R.; Finke, M.; Murata, Noboru; Schulten, K.; Amari, S.

In: Neural Computation, Vol. 8, No. 5, 01.07.1996, p. 1085-1106.

Research output: Contribution to journalArticle

Müller, KR, Finke, M, Murata, N, Schulten, K & Amari, S 1996, 'A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks', Neural Computation, vol. 8, no. 5, pp. 1085-1106.
Müller, K. R. ; Finke, M. ; Murata, Noboru ; Schulten, K. ; Amari, S. / A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks. In: Neural Computation. 1996 ; Vol. 8, No. 5. pp. 1085-1106.
@article{c28c363ce73f454ea64dff617f3cbec5,
title = "A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks",
abstract = "The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.",
author = "M{\"u}ller, {K. R.} and M. Finke and Noboru Murata and K. Schulten and S. Amari",
year = "1996",
month = "7",
day = "1",
language = "English",
volume = "8",
pages = "1085--1106",
journal = "Neural Computation",
issn = "0899-7667",
publisher = "MIT Press Journals",
number = "5",

}

TY - JOUR

T1 - A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks

AU - Müller, K. R.

AU - Finke, M.

AU - Murata, Noboru

AU - Schulten, K.

AU - Amari, S.

PY - 1996/7/1

Y1 - 1996/7/1

N2 - The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.

AB - The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.

UR - http://www.scopus.com/inward/record.url?scp=0030188747&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030188747&partnerID=8YFLogxK

M3 - Article

VL - 8

SP - 1085

EP - 1106

JO - Neural Computation

JF - Neural Computation

SN - 0899-7667

IS - 5

ER -