Sampling hidden parameters from oracle distribution

Sho Sonoda, Noboru Murata

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    3 Citations (Scopus)

    Abstract

    A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.

    Original languageEnglish
    Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    PublisherSpringer Verlag
    Pages539-546
    Number of pages8
    Volume8681 LNCS
    ISBN (Print)9783319111780
    DOIs
    Publication statusPublished - 2014
    Event24th International Conference on Artificial Neural Networks, ICANN 2014 - Hamburg, Germany
    Duration: 2014 Sep 152014 Sep 19

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume8681 LNCS
    ISSN (Print)03029743
    ISSN (Electronic)16113349

    Other

    Other24th International Conference on Artificial Neural Networks, ICANN 2014
    CountryGermany
    CityHamburg
    Period14/9/1514/9/19

    Fingerprint

    Sampling
    Neural networks
    Normal distribution
    Neural Networks
    Backpropagation
    Probability distributions
    Back Propagation
    Integral Representation
    Gaussian distribution
    Linear Time
    Probability Distribution
    Numerical Experiment
    Experiments
    Learning

    Keywords

    • backpropagation
    • Integral representation
    • neural networks
    • oracle distribution
    • sampling learning
    • weight initialization

    ASJC Scopus subject areas

    • Computer Science(all)
    • Theoretical Computer Science

    Cite this

    Sonoda, S., & Murata, N. (2014). Sampling hidden parameters from oracle distribution. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8681 LNCS, pp. 539-546). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8681 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-11179-7_68

    Sampling hidden parameters from oracle distribution. / Sonoda, Sho; Murata, Noboru.

    Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8681 LNCS Springer Verlag, 2014. p. 539-546 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8681 LNCS).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Sonoda, S & Murata, N 2014, Sampling hidden parameters from oracle distribution. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 8681 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8681 LNCS, Springer Verlag, pp. 539-546, 24th International Conference on Artificial Neural Networks, ICANN 2014, Hamburg, Germany, 14/9/15. https://doi.org/10.1007/978-3-319-11179-7_68
    Sonoda S, Murata N. Sampling hidden parameters from oracle distribution. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8681 LNCS. Springer Verlag. 2014. p. 539-546. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-11179-7_68
    Sonoda, Sho ; Murata, Noboru. / Sampling hidden parameters from oracle distribution. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8681 LNCS Springer Verlag, 2014. pp. 539-546 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
    @inproceedings{9dcf034f1cba48e886c9d62855f72809,
    title = "Sampling hidden parameters from oracle distribution",
    abstract = "A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.",
    keywords = "backpropagation, Integral representation, neural networks, oracle distribution, sampling learning, weight initialization",
    author = "Sho Sonoda and Noboru Murata",
    year = "2014",
    doi = "10.1007/978-3-319-11179-7_68",
    language = "English",
    isbn = "9783319111780",
    volume = "8681 LNCS",
    series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
    publisher = "Springer Verlag",
    pages = "539--546",
    booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

    }

    TY - GEN

    T1 - Sampling hidden parameters from oracle distribution

    AU - Sonoda, Sho

    AU - Murata, Noboru

    PY - 2014

    Y1 - 2014

    N2 - A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.

    AB - A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.

    KW - backpropagation

    KW - Integral representation

    KW - neural networks

    KW - oracle distribution

    KW - sampling learning

    KW - weight initialization

    UR - http://www.scopus.com/inward/record.url?scp=84958526124&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84958526124&partnerID=8YFLogxK

    U2 - 10.1007/978-3-319-11179-7_68

    DO - 10.1007/978-3-319-11179-7_68

    M3 - Conference contribution

    SN - 9783319111780

    VL - 8681 LNCS

    T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    SP - 539

    EP - 546

    BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    PB - Springer Verlag

    ER -