TY - GEN
T1 - Evaluating search result diversity using intent hierarchies
AU - Wang, Xiaojie
AU - Dou, Zhicheng
AU - Sakai, Tetsuya
AU - Wen, Ji Rong
N1 - Funding Information:
This work was supported by the National Key Basic Research Program (973 Program) of China under grant No. 2014CB340403, and the Fundamental Research Funds for the Central Universities, the Research Funds of Renmin University of China No. 15XNLF03, the National Natural Science Foundation of China (Grant No. 61502501, 61502502, and 61502503)
Publisher Copyright:
© 2016 ACM.
PY - 2016/7/7
Y1 - 2016/7/7
N2 - Search result diversification aims at returning diversified document lists to cover different user intents for ambiguous or broad queries. Existing diversity measures assume that user intents are independent or exclusive, and do not consider the relationships among the intents. In this paper, we introduce intent hierarchies to model the relationships among intents. Based on intent hierarchies, we propose several hierarchical measures that can consider the relationships among intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections. Our main experimental findings are: (1) Hierarchical measures are generally more discriminative and intuitive than existing measures using flat lists of intents; (2) When the queries have multilayer intent hierarchies, hierarchical measures are less correlated to existing measures, but can get more improvement in discriminative power; (3) Hierarchical measures are more intuitive in terms of diversity or relevance. The hierarchical measures using the whole intent hierarchies are more intuitive than only using the leaf nodes in terms of diversity and relevance.
AB - Search result diversification aims at returning diversified document lists to cover different user intents for ambiguous or broad queries. Existing diversity measures assume that user intents are independent or exclusive, and do not consider the relationships among the intents. In this paper, we introduce intent hierarchies to model the relationships among intents. Based on intent hierarchies, we propose several hierarchical measures that can consider the relationships among intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections. Our main experimental findings are: (1) Hierarchical measures are generally more discriminative and intuitive than existing measures using flat lists of intents; (2) When the queries have multilayer intent hierarchies, hierarchical measures are less correlated to existing measures, but can get more improvement in discriminative power; (3) Hierarchical measures are more intuitive in terms of diversity or relevance. The hierarchical measures using the whole intent hierarchies are more intuitive than only using the leaf nodes in terms of diversity and relevance.
KW - Ambiguity
KW - Diversity
KW - Evaluation
KW - Hierarchy
KW - Novelty
UR - http://www.scopus.com/inward/record.url?scp=84980324135&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84980324135&partnerID=8YFLogxK
U2 - 10.1145/2911451.2911497
DO - 10.1145/2911451.2911497
M3 - Conference contribution
AN - SCOPUS:84980324135
T3 - SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval
SP - 415
EP - 424
BT - SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval
PB - Association for Computing Machinery, Inc
T2 - 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016
Y2 - 17 July 2016 through 21 July 2016
ER -