A conditional entropy minimization criterion for dimensionality reduction and multiple kernel learning

Hideitsu Hino, Noboru Murata

    Research output: Contribution to journalArticle

    18 Citations (Scopus)

    Abstract

    Reducing the dimensionality of high-dimensional data without losing its essential information is an important task in information processing. When class labels of training data are available, Fisher discriminant analysis (FDA) has been widely used. However, the optimality of FDA is guaranteed only in a very restricted ideal circumstance, and it is often observed that FDA does not provide a good classification surface for many real problems. This letter treats the problem of supervised dimensionality reduction from the viewpoint of information theory and proposes a framework of dimensionality reduction based on classconditional entropy minimization. The proposed linear dimensionalityreduction technique is validated both theoretically and experimentally. Then, through kernel Fisher discriminant analysis (KFDA), the multiple kernel learning problem is treated in the proposed framework, and a novel algorithm, which iteratively optimizes the parameters of the classification function and kernel combination coefficients, is proposed. The algorithm is experimentally shown to be comparable to or outperforms KFDA for large-scale benchmark data sets, and comparable to other multiple kernel learning techniques on the yeast protein function annotation task.

    Original languageEnglish
    Pages (from-to)2887-2923
    Number of pages37
    JournalNeural Computation
    Volume22
    Issue number11
    DOIs
    Publication statusPublished - 2010 Nov

    Fingerprint

    Entropy
    Discriminant Analysis
    Learning
    Information Theory
    Molecular Sequence Annotation
    Benchmarking
    Fungal Proteins
    Automatic Data Processing
    Kernel

    ASJC Scopus subject areas

    • Cognitive Neuroscience

    Cite this

    A conditional entropy minimization criterion for dimensionality reduction and multiple kernel learning. / Hino, Hideitsu; Murata, Noboru.

    In: Neural Computation, Vol. 22, No. 11, 11.2010, p. 2887-2923.

    Research output: Contribution to journalArticle

    @article{de30b95a5576423fa18c85f71c066fdc,
    title = "A conditional entropy minimization criterion for dimensionality reduction and multiple kernel learning",
    abstract = "Reducing the dimensionality of high-dimensional data without losing its essential information is an important task in information processing. When class labels of training data are available, Fisher discriminant analysis (FDA) has been widely used. However, the optimality of FDA is guaranteed only in a very restricted ideal circumstance, and it is often observed that FDA does not provide a good classification surface for many real problems. This letter treats the problem of supervised dimensionality reduction from the viewpoint of information theory and proposes a framework of dimensionality reduction based on classconditional entropy minimization. The proposed linear dimensionalityreduction technique is validated both theoretically and experimentally. Then, through kernel Fisher discriminant analysis (KFDA), the multiple kernel learning problem is treated in the proposed framework, and a novel algorithm, which iteratively optimizes the parameters of the classification function and kernel combination coefficients, is proposed. The algorithm is experimentally shown to be comparable to or outperforms KFDA for large-scale benchmark data sets, and comparable to other multiple kernel learning techniques on the yeast protein function annotation task.",
    author = "Hideitsu Hino and Noboru Murata",
    year = "2010",
    month = "11",
    doi = "10.1162/NECO_a_00027",
    language = "English",
    volume = "22",
    pages = "2887--2923",
    journal = "Neural Computation",
    issn = "0899-7667",
    publisher = "MIT Press Journals",
    number = "11",

    }

    TY - JOUR

    T1 - A conditional entropy minimization criterion for dimensionality reduction and multiple kernel learning

    AU - Hino, Hideitsu

    AU - Murata, Noboru

    PY - 2010/11

    Y1 - 2010/11

    N2 - Reducing the dimensionality of high-dimensional data without losing its essential information is an important task in information processing. When class labels of training data are available, Fisher discriminant analysis (FDA) has been widely used. However, the optimality of FDA is guaranteed only in a very restricted ideal circumstance, and it is often observed that FDA does not provide a good classification surface for many real problems. This letter treats the problem of supervised dimensionality reduction from the viewpoint of information theory and proposes a framework of dimensionality reduction based on classconditional entropy minimization. The proposed linear dimensionalityreduction technique is validated both theoretically and experimentally. Then, through kernel Fisher discriminant analysis (KFDA), the multiple kernel learning problem is treated in the proposed framework, and a novel algorithm, which iteratively optimizes the parameters of the classification function and kernel combination coefficients, is proposed. The algorithm is experimentally shown to be comparable to or outperforms KFDA for large-scale benchmark data sets, and comparable to other multiple kernel learning techniques on the yeast protein function annotation task.

    AB - Reducing the dimensionality of high-dimensional data without losing its essential information is an important task in information processing. When class labels of training data are available, Fisher discriminant analysis (FDA) has been widely used. However, the optimality of FDA is guaranteed only in a very restricted ideal circumstance, and it is often observed that FDA does not provide a good classification surface for many real problems. This letter treats the problem of supervised dimensionality reduction from the viewpoint of information theory and proposes a framework of dimensionality reduction based on classconditional entropy minimization. The proposed linear dimensionalityreduction technique is validated both theoretically and experimentally. Then, through kernel Fisher discriminant analysis (KFDA), the multiple kernel learning problem is treated in the proposed framework, and a novel algorithm, which iteratively optimizes the parameters of the classification function and kernel combination coefficients, is proposed. The algorithm is experimentally shown to be comparable to or outperforms KFDA for large-scale benchmark data sets, and comparable to other multiple kernel learning techniques on the yeast protein function annotation task.

    UR - http://www.scopus.com/inward/record.url?scp=78149311149&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=78149311149&partnerID=8YFLogxK

    U2 - 10.1162/NECO_a_00027

    DO - 10.1162/NECO_a_00027

    M3 - Article

    C2 - 20804381

    AN - SCOPUS:78149311149

    VL - 22

    SP - 2887

    EP - 2923

    JO - Neural Computation

    JF - Neural Computation

    SN - 0899-7667

    IS - 11

    ER -