Search Result Diversity Evaluation based on Intent Hierarchies

Xiaojie Wang, Ji Rong Wen, Zhicheng Dou, Tetsuya Sakai, Rui Zhang

    Research output: Contribution to journalArticle

    1 Citation (Scopus)

    Abstract

    Search result diversification aims at returning diversified document lists to cover different user intents of a query. Existing diversity measures assume that the intents of a query are disjoint, and do not consider their relationships. In this paper, we introduce intent hierarchies to model the relationships between intents, and present four weighing schemes. Based on intent hierarchies, we propose several hierarchical measures that take into account the relationships between intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections and by using NTCIR-11 IMine test collection. Our main experimental findings are: (1) Hierarchical measures are more discriminative and intuitive than existing measures. In terms of intuitiveness, it is preferable for hierarchical measures to use the whole intent hierarchies than to use only the leaf nodes; (2) The types of intent hierarchies used affect the discriminative power and intuitiveness of hierarchical measures. We suggest the best type of intent hierarchies to be used according to whether the nonuniform weights are available; (3) To measure the benefits of the diversification algorithms which use automatically mined hierarchical intents, it is important to use hierarchical measures instead of existing measures.

    Original languageEnglish
    JournalIEEE Transactions on Knowledge and Data Engineering
    DOIs
    Publication statusAccepted/In press - 2017 Jul 20

    Fingerprint

    Weighing

    Keywords

    • Ambiguity
    • Companies
    • Diversity
    • Evaluation
    • Gain measurement
    • Gold
    • Hierarchy
    • Novelty
    • Power measurement
    • Q measurement
    • Standards
    • Weight measurement

    ASJC Scopus subject areas

    • Information Systems
    • Computer Science Applications
    • Computational Theory and Mathematics

    Cite this

    Search Result Diversity Evaluation based on Intent Hierarchies. / Wang, Xiaojie; Wen, Ji Rong; Dou, Zhicheng; Sakai, Tetsuya; Zhang, Rui.

    In: IEEE Transactions on Knowledge and Data Engineering, 20.07.2017.

    Research output: Contribution to journalArticle

    @article{af8b18596ba84d9998d7c363df8d40c9,
    title = "Search Result Diversity Evaluation based on Intent Hierarchies",
    abstract = "Search result diversification aims at returning diversified document lists to cover different user intents of a query. Existing diversity measures assume that the intents of a query are disjoint, and do not consider their relationships. In this paper, we introduce intent hierarchies to model the relationships between intents, and present four weighing schemes. Based on intent hierarchies, we propose several hierarchical measures that take into account the relationships between intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections and by using NTCIR-11 IMine test collection. Our main experimental findings are: (1) Hierarchical measures are more discriminative and intuitive than existing measures. In terms of intuitiveness, it is preferable for hierarchical measures to use the whole intent hierarchies than to use only the leaf nodes; (2) The types of intent hierarchies used affect the discriminative power and intuitiveness of hierarchical measures. We suggest the best type of intent hierarchies to be used according to whether the nonuniform weights are available; (3) To measure the benefits of the diversification algorithms which use automatically mined hierarchical intents, it is important to use hierarchical measures instead of existing measures.",
    keywords = "Ambiguity, Companies, Diversity, Evaluation, Gain measurement, Gold, Hierarchy, Novelty, Power measurement, Q measurement, Standards, Weight measurement",
    author = "Xiaojie Wang and Wen, {Ji Rong} and Zhicheng Dou and Tetsuya Sakai and Rui Zhang",
    year = "2017",
    month = "7",
    day = "20",
    doi = "10.1109/TKDE.2017.2729559",
    language = "English",
    journal = "IEEE Transactions on Knowledge and Data Engineering",
    issn = "1041-4347",
    publisher = "IEEE Computer Society",

    }

    TY - JOUR

    T1 - Search Result Diversity Evaluation based on Intent Hierarchies

    AU - Wang, Xiaojie

    AU - Wen, Ji Rong

    AU - Dou, Zhicheng

    AU - Sakai, Tetsuya

    AU - Zhang, Rui

    PY - 2017/7/20

    Y1 - 2017/7/20

    N2 - Search result diversification aims at returning diversified document lists to cover different user intents of a query. Existing diversity measures assume that the intents of a query are disjoint, and do not consider their relationships. In this paper, we introduce intent hierarchies to model the relationships between intents, and present four weighing schemes. Based on intent hierarchies, we propose several hierarchical measures that take into account the relationships between intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections and by using NTCIR-11 IMine test collection. Our main experimental findings are: (1) Hierarchical measures are more discriminative and intuitive than existing measures. In terms of intuitiveness, it is preferable for hierarchical measures to use the whole intent hierarchies than to use only the leaf nodes; (2) The types of intent hierarchies used affect the discriminative power and intuitiveness of hierarchical measures. We suggest the best type of intent hierarchies to be used according to whether the nonuniform weights are available; (3) To measure the benefits of the diversification algorithms which use automatically mined hierarchical intents, it is important to use hierarchical measures instead of existing measures.

    AB - Search result diversification aims at returning diversified document lists to cover different user intents of a query. Existing diversity measures assume that the intents of a query are disjoint, and do not consider their relationships. In this paper, we introduce intent hierarchies to model the relationships between intents, and present four weighing schemes. Based on intent hierarchies, we propose several hierarchical measures that take into account the relationships between intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections and by using NTCIR-11 IMine test collection. Our main experimental findings are: (1) Hierarchical measures are more discriminative and intuitive than existing measures. In terms of intuitiveness, it is preferable for hierarchical measures to use the whole intent hierarchies than to use only the leaf nodes; (2) The types of intent hierarchies used affect the discriminative power and intuitiveness of hierarchical measures. We suggest the best type of intent hierarchies to be used according to whether the nonuniform weights are available; (3) To measure the benefits of the diversification algorithms which use automatically mined hierarchical intents, it is important to use hierarchical measures instead of existing measures.

    KW - Ambiguity

    KW - Companies

    KW - Diversity

    KW - Evaluation

    KW - Gain measurement

    KW - Gold

    KW - Hierarchy

    KW - Novelty

    KW - Power measurement

    KW - Q measurement

    KW - Standards

    KW - Weight measurement

    UR - http://www.scopus.com/inward/record.url?scp=85028926924&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85028926924&partnerID=8YFLogxK

    U2 - 10.1109/TKDE.2017.2729559

    DO - 10.1109/TKDE.2017.2729559

    M3 - Article

    JO - IEEE Transactions on Knowledge and Data Engineering

    JF - IEEE Transactions on Knowledge and Data Engineering

    SN - 1041-4347

    ER -