On the reliability of information retrieval metrics based on graded relevance

Research output: Contribution to journalArticle

63 Citations (Scopus)

Abstract

This paper compares 14 information retrieval metrics based on graded relevance, together with 10 traditional metrics based on binary relevance, in terms of stability, sensitivity and resemblance of system rankings. More specifically, we compare these metrics using the Buckley/Voorhees stability method, the Voorhees/Buckley swap method and Kendall's rank correlation, with three data sets comprising test collections and submitted runs from NTCIR. Our experiments show that (Average) Normalised Discounted Cumulative Gain at document cut-off l are the best among the rank-based graded-relevance metrics, provided that l is large. On the other hand, if one requires a recall-based graded-relevance metric that is highly correlated with Average Precision, then Q-measure is the best choice. Moreover, these best graded-relevance metrics are at least as stable and sensitive as Average Precision, and are fairly robust to the choice of gain values.

Original languageEnglish
Pages (from-to)531-548
Number of pages18
JournalInformation Processing and Management
Volume43
Issue number2
DOIs
Publication statusPublished - 2007 Mar 1
Externally publishedYes

Keywords

  • Cumulative gain
  • Evaluation
  • Graded relevance
  • Q-measure
  • Reliability

ASJC Scopus subject areas

  • Information Systems
  • Media Technology
  • Computer Science Applications
  • Management Science and Operations Research
  • Library and Information Sciences

Fingerprint Dive into the research topics of 'On the reliability of information retrieval metrics based on graded relevance'. Together they form a unique fingerprint.

  • Cite this