A study on difference of codelengths between codes based on MDL principle and bayes codes for given prior distributions

Masayuki Goto, Toshiyasu Matsushima, Shigeichi Hirasawa

    Research output: Contribution to journalArticle

    Abstract

    The principle of the Minimum Description Length (MDL) proposed by J. Rissanen provides a type of structure for the model estimation based on probabilistic model selection allowing minimization of the codelength. On the other hand, the use of Bayes codes makes it possible to find a coding function from a mix of probabilistic models without specifying any concrete model. It has been pointed out that codes based on the MDL principle (MDL codes) are closely related to Bayes theory because in the definition of the description length of the probabilistic model, an unknown prior distribution is assumed. In this paper, we apply asymptotic analysis to the codelength difference between the MDL codes and Bayes codes, including cases of different prior distributions. The results of the analysis clearly show that in the case of discrete model families, codes having a high prior distribution in true models (that is, the models for which an advantageous prior distribution is assumed) are favorable, but in the case of parametric model families, Bayes codes have shorter codelength than the MDL codes even in the cases of advantageous prior distribution assumed for the MDL codes.

    Original languageEnglish
    Pages (from-to)30-40
    Number of pages11
    JournalElectronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)
    Volume84
    Issue number4
    Publication statusPublished - 2001

    Fingerprint

    Asymptotic analysis
    Statistical Models

    Keywords

    • Bayes code
    • Information source coding
    • MDL principle
    • Prior distribution

    ASJC Scopus subject areas

    • Electrical and Electronic Engineering
    • Computer Networks and Communications

    Cite this

    @article{c94dca70856a437e9fa94b575e37605e,
    title = "A study on difference of codelengths between codes based on MDL principle and bayes codes for given prior distributions",
    abstract = "The principle of the Minimum Description Length (MDL) proposed by J. Rissanen provides a type of structure for the model estimation based on probabilistic model selection allowing minimization of the codelength. On the other hand, the use of Bayes codes makes it possible to find a coding function from a mix of probabilistic models without specifying any concrete model. It has been pointed out that codes based on the MDL principle (MDL codes) are closely related to Bayes theory because in the definition of the description length of the probabilistic model, an unknown prior distribution is assumed. In this paper, we apply asymptotic analysis to the codelength difference between the MDL codes and Bayes codes, including cases of different prior distributions. The results of the analysis clearly show that in the case of discrete model families, codes having a high prior distribution in true models (that is, the models for which an advantageous prior distribution is assumed) are favorable, but in the case of parametric model families, Bayes codes have shorter codelength than the MDL codes even in the cases of advantageous prior distribution assumed for the MDL codes.",
    keywords = "Bayes code, Information source coding, MDL principle, Prior distribution",
    author = "Masayuki Goto and Toshiyasu Matsushima and Shigeichi Hirasawa",
    year = "2001",
    language = "English",
    volume = "84",
    pages = "30--40",
    journal = "Electronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)",
    issn = "1042-0967",
    publisher = "John Wiley and Sons Inc.",
    number = "4",

    }

    TY - JOUR

    T1 - A study on difference of codelengths between codes based on MDL principle and bayes codes for given prior distributions

    AU - Goto, Masayuki

    AU - Matsushima, Toshiyasu

    AU - Hirasawa, Shigeichi

    PY - 2001

    Y1 - 2001

    N2 - The principle of the Minimum Description Length (MDL) proposed by J. Rissanen provides a type of structure for the model estimation based on probabilistic model selection allowing minimization of the codelength. On the other hand, the use of Bayes codes makes it possible to find a coding function from a mix of probabilistic models without specifying any concrete model. It has been pointed out that codes based on the MDL principle (MDL codes) are closely related to Bayes theory because in the definition of the description length of the probabilistic model, an unknown prior distribution is assumed. In this paper, we apply asymptotic analysis to the codelength difference between the MDL codes and Bayes codes, including cases of different prior distributions. The results of the analysis clearly show that in the case of discrete model families, codes having a high prior distribution in true models (that is, the models for which an advantageous prior distribution is assumed) are favorable, but in the case of parametric model families, Bayes codes have shorter codelength than the MDL codes even in the cases of advantageous prior distribution assumed for the MDL codes.

    AB - The principle of the Minimum Description Length (MDL) proposed by J. Rissanen provides a type of structure for the model estimation based on probabilistic model selection allowing minimization of the codelength. On the other hand, the use of Bayes codes makes it possible to find a coding function from a mix of probabilistic models without specifying any concrete model. It has been pointed out that codes based on the MDL principle (MDL codes) are closely related to Bayes theory because in the definition of the description length of the probabilistic model, an unknown prior distribution is assumed. In this paper, we apply asymptotic analysis to the codelength difference between the MDL codes and Bayes codes, including cases of different prior distributions. The results of the analysis clearly show that in the case of discrete model families, codes having a high prior distribution in true models (that is, the models for which an advantageous prior distribution is assumed) are favorable, but in the case of parametric model families, Bayes codes have shorter codelength than the MDL codes even in the cases of advantageous prior distribution assumed for the MDL codes.

    KW - Bayes code

    KW - Information source coding

    KW - MDL principle

    KW - Prior distribution

    UR - http://www.scopus.com/inward/record.url?scp=0035127714&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=0035127714&partnerID=8YFLogxK

    M3 - Article

    VL - 84

    SP - 30

    EP - 40

    JO - Electronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)

    JF - Electronics and Communications in Japan, Part III: Fundamental Electronic Science (English translation of Denshi Tsushin Gakkai Ronbunshi)

    SN - 1042-0967

    IS - 4

    ER -