Non-parametric entropy estimators based on simple linear regression

Hideitsu Hino, Kensuke Koshijima, Noboru Murata

    Research output: Contribution to journalArticlepeer-review

    6 Citations (Scopus)


    Abstract Estimators for differential entropy are proposed. The estimators are based on the second order expansion of the probability mass around the inspection point with respect to the distance from the point. Simple linear regression is utilized to estimate the values of density function and its second derivative at a point. After estimating the values of the probability density function at each of the given sample points, by taking the empirical average of the negative logarithm of the density estimates, two entropy estimators are derived. Other entropy estimators which directly estimate entropy by linear regression, are also proposed. The proposed four estimators are shown to perform well through numerical experiments for various probability distributions.

    Original languageEnglish
    Article number6063
    Pages (from-to)72-84
    Number of pages13
    JournalComputational Statistics and Data Analysis
    Publication statusPublished - 2015


    • Entropy estimation
    • Non-parametric
    • Simple linear regression

    ASJC Scopus subject areas

    • Computational Mathematics
    • Computational Theory and Mathematics
    • Statistics and Probability
    • Applied Mathematics

    Fingerprint Dive into the research topics of 'Non-parametric entropy estimators based on simple linear regression'. Together they form a unique fingerprint.

    Cite this