Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC

Hyeyoung Park, Noboru Murata, Shun Ichi Amari

    Research output: Contribution to journalArticle

    13 Citations (Scopus)

    Abstract

    Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly, the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

    Original languageEnglish
    Pages (from-to)355-382
    Number of pages28
    JournalNeural Computation
    Volume16
    Issue number2
    DOIs
    Publication statusPublished - 2004 Feb

    Fingerprint

    Gradient methods
    Computational efficiency
    Learning
    Neural networks
    Computer simulation
    Experiments
    Benchmarking
    Information Services
    Computer Simulation
    Efficiency

    ASJC Scopus subject areas

    • Control and Systems Engineering
    • Artificial Intelligence
    • Neuroscience(all)

    Cite this

    Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC. / Park, Hyeyoung; Murata, Noboru; Amari, Shun Ichi.

    In: Neural Computation, Vol. 16, No. 2, 02.2004, p. 355-382.

    Research output: Contribution to journalArticle

    @article{d68cc28df8734ed480300f5fd092571b,
    title = "Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC",
    abstract = "Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly, the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.",
    author = "Hyeyoung Park and Noboru Murata and Amari, {Shun Ichi}",
    year = "2004",
    month = "2",
    doi = "10.1162/089976604322742065",
    language = "English",
    volume = "16",
    pages = "355--382",
    journal = "Neural Computation",
    issn = "0899-7667",
    publisher = "MIT Press Journals",
    number = "2",

    }

    TY - JOUR

    T1 - Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC

    AU - Park, Hyeyoung

    AU - Murata, Noboru

    AU - Amari, Shun Ichi

    PY - 2004/2

    Y1 - 2004/2

    N2 - Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly, the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

    AB - Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly, the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.

    UR - http://www.scopus.com/inward/record.url?scp=0347029918&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=0347029918&partnerID=8YFLogxK

    U2 - 10.1162/089976604322742065

    DO - 10.1162/089976604322742065

    M3 - Article

    C2 - 15006100

    AN - SCOPUS:0347029918

    VL - 16

    SP - 355

    EP - 382

    JO - Neural Computation

    JF - Neural Computation

    SN - 0899-7667

    IS - 2

    ER -