Non-parametric entropy estimators based on simple linear regression

Hideitsu Hino, Kensuke Koshijima, Noboru Murata

    Research output: Contribution to journalArticle

    4 Citations (Scopus)

    Abstract

    Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.

    Original languageEnglish
    Article number6063
    Pages (from-to)72-84
    Number of pages13
    JournalComputational Statistics and Data Analysis
    Volume89
    DOIs
    Publication statusPublished - 2015

    Fingerprint

    Simple Linear Regression
    Linear regression
    Entropy
    Estimator
    Probability density function
    Density Estimates
    Sample point
    Probability distributions
    Second derivative
    Logarithm
    Density Function
    Inspection
    Estimate
    Derivatives
    Probability Distribution
    Numerical Experiment
    Experiments

    Keywords

    • Entropy estimation
    • Non-parametric
    • Simple linear regression

    ASJC Scopus subject areas

    • Computational Mathematics
    • Computational Theory and Mathematics
    • Statistics and Probability
    • Applied Mathematics

    Cite this

    Non-parametric entropy estimators based on simple linear regression. / Hino, Hideitsu; Koshijima, Kensuke; Murata, Noboru.

    In: Computational Statistics and Data Analysis, Vol. 89, 6063, 2015, p. 72-84.

    Research output: Contribution to journalArticle

    @article{c2c6b6c97b70444e9604237f2ae569a1,
    title = "Non-parametric entropy estimators based on simple linear regression",
    abstract = "Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.",
    keywords = "Entropy estimation, Non-parametric, Simple linear regression",
    author = "Hideitsu Hino and Kensuke Koshijima and Noboru Murata",
    year = "2015",
    doi = "10.1016/j.csda.2015.03.011",
    language = "English",
    volume = "89",
    pages = "72--84",
    journal = "Computational Statistics and Data Analysis",
    issn = "0167-9473",
    publisher = "Elsevier",

    }

    TY - JOUR

    T1 - Non-parametric entropy estimators based on simple linear regression

    AU - Hino, Hideitsu

    AU - Koshijima, Kensuke

    AU - Murata, Noboru

    PY - 2015

    Y1 - 2015

    N2 - Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.

    AB - Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.

    KW - Entropy estimation

    KW - Non-parametric

    KW - Simple linear regression

    UR - http://www.scopus.com/inward/record.url?scp=84926286540&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84926286540&partnerID=8YFLogxK

    U2 - 10.1016/j.csda.2015.03.011

    DO - 10.1016/j.csda.2015.03.011

    M3 - Article

    VL - 89

    SP - 72

    EP - 84

    JO - Computational Statistics and Data Analysis

    JF - Computational Statistics and Data Analysis

    SN - 0167-9473

    M1 - 6063

    ER -