Alternatives to Bpref

Research output: Chapter in Book/Report/Conference proceedingConference contribution

86 Citations (Scopus)

Abstract

Recently, a number of TREC tracks have adopted a retrieval effectiveness metric called bpref which has been designed for evaluation environments with incomplete relevance data. A graded-relevance version of this metric called rpref has also been proposed. However, we show that the application of Q-measure, normalised Discounted Cumulative Gain (nDCG) or Average Precision (AveP)to condensed lists, obtained by ?ltering out all unjudged documents from the original ranked lists, is actually a better solution to the incompleteness problem than bpref. Furthermore, we show that the use of graded relevance boosts the robustness of IR evaluation to incompleteness and therefore that Q-measure and nDCG based on condensed lists are the best choices. To this end, we use four graded-relevance test collections from NTCIR to compare ten different IR metrics in terms of system ranking stability and pairwise discriminative power.

Original languageEnglish
Title of host publicationProceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07
Pages71-78
Number of pages8
DOIs
Publication statusPublished - 2007 Nov 30
Event30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07 - Amsterdam, Netherlands
Duration: 2007 Jul 232007 Jul 27

Publication series

NameProceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Conference

Conference30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07
CountryNetherlands
CityAmsterdam
Period07/7/2307/7/27

    Fingerprint

Keywords

  • Evaluation metrics
  • Graded relevance
  • Test collection

ASJC Scopus subject areas

  • Information Systems
  • Software
  • Applied Mathematics

Cite this

Sakai, T. (2007). Alternatives to Bpref. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07 (pp. 71-78). (Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07). https://doi.org/10.1145/1277741.1277756