On the reliability of information retrieval metrics based on graded relevance

Tetsuya Sakai*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

72 Citations (Scopus)


This paper compares 14 information retrieval metrics based on graded relevance, together with 10 traditional metrics based on binary relevance, in terms of stability, sensitivity and resemblance of system rankings. More specifically, we compare these metrics using the Buckley/Voorhees stability method, the Voorhees/Buckley swap method and Kendall's rank correlation, with three data sets comprising test collections and submitted runs from NTCIR. Our experiments show that (Average) Normalised Discounted Cumulative Gain at document cut-off l are the best among the rank-based graded-relevance metrics, provided that l is large. On the other hand, if one requires a recall-based graded-relevance metric that is highly correlated with Average Precision, then Q-measure is the best choice. Moreover, these best graded-relevance metrics are at least as stable and sensitive as Average Precision, and are fairly robust to the choice of gain values.

Original languageEnglish
Pages (from-to)531-548
Number of pages18
JournalInformation Processing and Management
Issue number2
Publication statusPublished - 2007 Mar
Externally publishedYes


  • Cumulative gain
  • Evaluation
  • Graded relevance
  • Q-measure
  • Reliability

ASJC Scopus subject areas

  • Information Systems
  • Media Technology
  • Computer Science Applications
  • Management Science and Operations Research
  • Library and Information Sciences


Dive into the research topics of 'On the reliability of information retrieval metrics based on graded relevance'. Together they form a unique fingerprint.

Cite this