How intuitive are diversified search metrics? Concordance test results for the diversity U-measures

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Most of the existing Information Retrieval (IR) metrics discount the value of each retrieved relevant document based on its rank. This statement also applies to the evaluation of diversified search: the widely-used diversity metrics, namely, α-nDCG, Intent-Aware Expected Reciprocal Rank (ERR-IA) and D#-nDCG, are all rank-based. These evaluation metrics regard the system output as a list of document IDs, and ignore all other features such as snippets and document full texts of various lengths. In contrast, the U-measure framework of Sakai and Dou uses the amount of text read by the user as the foundation for discounting the value of relevant information, and can take into account the user's snippet reading and full text reading behaviours. The present study compares the diversity versions of U-measure (D-U and U-IA) with the state-of-the-art diversity metrics using the concordance test: given a pair of ranked lists, we quantify the ability of each metric to favour the more diversified and more relevant list. Our results show that while D#-nDCG is the overall winner in terms of simultaneous concordance with diversity and relevance, D-U and U-IA statistically significantly outperform other state-of-the-art metrics. Moreover, in terms of concordance with relevance alone, D-U and U-IA significantly outperform all rank-based diversity metrics. Thus, D-U and U-IA are not only more realistic but also more relevance-oriented than other diversity metrics.

Original languageEnglish
Title of host publicationInformation Retrieval Technology - 9th Asia Information Retrieval Societies Conference, AIRS 2013, Proceedings
Pages13-24
Number of pages12
DOIs
Publication statusPublished - 2013 Dec 1
Event9th Asia Information Retrieval Societies Conference on Information Retrieval Technology, AIRS 2013 - Singapore, Singapore
Duration: 2013 Dec 92013 Dec 11

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8281 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference9th Asia Information Retrieval Societies Conference on Information Retrieval Technology, AIRS 2013
CountrySingapore
CitySingapore
Period13/12/913/12/11

    Fingerprint

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Sakai, T. (2013). How intuitive are diversified search metrics? Concordance test results for the diversity U-measures. In Information Retrieval Technology - 9th Asia Information Retrieval Societies Conference, AIRS 2013, Proceedings (pp. 13-24). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8281 LNCS). https://doi.org/10.1007/978-3-642-45068-6_2