A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks

K. R. Müller, M. Finke, Noboru Murata, K. Schulten, S. Amari

Research output: Contribution to journalArticle

26 Citations (Scopus)

Abstract

The universal asymptotic scaling laws proposed by Amari et al. are studied in large scale simulations using a CM5. Small stochastic multilayer feedforward networks trained with backpropagation are investigated. In the range of a large number of training patterns t, the asymptotic generalization error scales as 1/t as predicted. For a medium range t a faster 1/t2 scaling is observed. This effect is explained by using higher order corrections of the likelihood expansion. It is shown for small t that the scaling law changes drastically, when the network undergoes a transition from strong overfitting to effective learning.

Original languageEnglish
Pages (from-to)1085-1106
Number of pages22
JournalNeural Computation
Volume8
Issue number5
Publication statusPublished - 1996 Jul 1
Externally publishedYes

    Fingerprint

ASJC Scopus subject areas

  • Artificial Intelligence
  • Control and Systems Engineering
  • Neuroscience(all)

Cite this

Müller, K. R., Finke, M., Murata, N., Schulten, K., & Amari, S. (1996). A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks. Neural Computation, 8(5), 1085-1106.