Statistical analysis of regularization constant from bayes, MDL and NIC Points of view

Shun Ichi Amari, Noboru Murata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

In order to avoid overfitting in neural learning, a regularization term is added to the loss function to be minimized. It is naturMly derived from the Bayesian standpoint. The present paper studies how to determine the regularization constant from the points of view of the empirical Bayes approach, the maximum description length (MDL) approach, and the network information criterion (NIC) approach. The asymptotic statistical analysis is given to elucidate their differences. These approaches are tightly connected with the method of model selection. The superiority of the NIC is shown from this analysis.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer Verlag
Pages284-293
Number of pages10
Volume1240 LNCS
ISBN (Print)3540630473, 9783540630470
Publication statusPublished - 1997
Externally publishedYes
Event4th International Work-Conference on Artificial and Natural Neural Networks, IWANN 1997 - Lanzarote, Canary Islands
Duration: 1997 Jun 41997 Jun 6

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume1240 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other4th International Work-Conference on Artificial and Natural Neural Networks, IWANN 1997
CityLanzarote, Canary Islands
Period97/6/497/6/6

Fingerprint

Information Criterion
Bayes
Statistical Analysis
Regularization
Statistical methods
Empirical Bayes
Overfitting
Loss Function
Asymptotic Analysis
Model Selection
Term
Learning

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Amari, S. I., & Murata, N. (1997). Statistical analysis of regularization constant from bayes, MDL and NIC Points of view. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 1240 LNCS, pp. 284-293). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 1240 LNCS). Springer Verlag.

Statistical analysis of regularization constant from bayes, MDL and NIC Points of view. / Amari, Shun Ichi; Murata, Noboru.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 1240 LNCS Springer Verlag, 1997. p. 284-293 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 1240 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Amari, SI & Murata, N 1997, Statistical analysis of regularization constant from bayes, MDL and NIC Points of view. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 1240 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 1240 LNCS, Springer Verlag, pp. 284-293, 4th International Work-Conference on Artificial and Natural Neural Networks, IWANN 1997, Lanzarote, Canary Islands, 97/6/4.
Amari SI, Murata N. Statistical analysis of regularization constant from bayes, MDL and NIC Points of view. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 1240 LNCS. Springer Verlag. 1997. p. 284-293. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
Amari, Shun Ichi ; Murata, Noboru. / Statistical analysis of regularization constant from bayes, MDL and NIC Points of view. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 1240 LNCS Springer Verlag, 1997. pp. 284-293 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{d50a3ec7aca7480eb126fc111ba9e8df,
title = "Statistical analysis of regularization constant from bayes, MDL and NIC Points of view",
abstract = "In order to avoid overfitting in neural learning, a regularization term is added to the loss function to be minimized. It is naturMly derived from the Bayesian standpoint. The present paper studies how to determine the regularization constant from the points of view of the empirical Bayes approach, the maximum description length (MDL) approach, and the network information criterion (NIC) approach. The asymptotic statistical analysis is given to elucidate their differences. These approaches are tightly connected with the method of model selection. The superiority of the NIC is shown from this analysis.",
author = "Amari, {Shun Ichi} and Noboru Murata",
year = "1997",
language = "English",
isbn = "3540630473",
volume = "1240 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "284--293",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Statistical analysis of regularization constant from bayes, MDL and NIC Points of view

AU - Amari, Shun Ichi

AU - Murata, Noboru

PY - 1997

Y1 - 1997

N2 - In order to avoid overfitting in neural learning, a regularization term is added to the loss function to be minimized. It is naturMly derived from the Bayesian standpoint. The present paper studies how to determine the regularization constant from the points of view of the empirical Bayes approach, the maximum description length (MDL) approach, and the network information criterion (NIC) approach. The asymptotic statistical analysis is given to elucidate their differences. These approaches are tightly connected with the method of model selection. The superiority of the NIC is shown from this analysis.

AB - In order to avoid overfitting in neural learning, a regularization term is added to the loss function to be minimized. It is naturMly derived from the Bayesian standpoint. The present paper studies how to determine the regularization constant from the points of view of the empirical Bayes approach, the maximum description length (MDL) approach, and the network information criterion (NIC) approach. The asymptotic statistical analysis is given to elucidate their differences. These approaches are tightly connected with the method of model selection. The superiority of the NIC is shown from this analysis.

UR - http://www.scopus.com/inward/record.url?scp=21744450484&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=21744450484&partnerID=8YFLogxK

M3 - Conference contribution

SN - 3540630473

SN - 9783540630470

VL - 1240 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 284

EP - 293

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

PB - Springer Verlag

ER -