How intuitive are diversified search metrics? Concordance test results for the diversity U-measures

研究成果: Conference contribution

5 被引用数 (Scopus)

抄録

Most of the existing Information Retrieval (IR) metrics discount the value of each retrieved relevant document based on its rank. This statement also applies to the evaluation of diversified search: the widely-used diversity metrics, namely, α-nDCG, Intent-Aware Expected Reciprocal Rank (ERR-IA) and D#-nDCG, are all rank-based. These evaluation metrics regard the system output as a list of document IDs, and ignore all other features such as snippets and document full texts of various lengths. In contrast, the U-measure framework of Sakai and Dou uses the amount of text read by the user as the foundation for discounting the value of relevant information, and can take into account the user's snippet reading and full text reading behaviours. The present study compares the diversity versions of U-measure (D-U and U-IA) with the state-of-the-art diversity metrics using the concordance test: given a pair of ranked lists, we quantify the ability of each metric to favour the more diversified and more relevant list. Our results show that while D#-nDCG is the overall winner in terms of simultaneous concordance with diversity and relevance, D-U and U-IA statistically significantly outperform other state-of-the-art metrics. Moreover, in terms of concordance with relevance alone, D-U and U-IA significantly outperform all rank-based diversity metrics. Thus, D-U and U-IA are not only more realistic but also more relevance-oriented than other diversity metrics.

本文言語English
ホスト出版物のタイトルInformation Retrieval Technology - 9th Asia Information Retrieval Societies Conference, AIRS 2013, Proceedings
ページ13-24
ページ数12
DOI
出版ステータスPublished - 2013 12 1
イベント9th Asia Information Retrieval Societies Conference on Information Retrieval Technology, AIRS 2013 - Singapore, Singapore
継続期間: 2013 12 92013 12 11

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
8281 LNCS
ISSN(印刷版)0302-9743
ISSN(電子版)1611-3349

Conference

Conference9th Asia Information Retrieval Societies Conference on Information Retrieval Technology, AIRS 2013
CountrySingapore
CitySingapore
Period13/12/913/12/11

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

フィンガープリント 「How intuitive are diversified search metrics? Concordance test results for the diversity U-measures」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル