On the reliability of information retrieval metrics based on graded relevance

Research output: Contribution to journalArticle

63 Citations (Scopus)


This paper compares 14 information retrieval metrics based on graded relevance, together with 10 traditional metrics based on binary relevance, in terms of stability, sensitivity and resemblance of system rankings. More specifically, we compare these metrics using the Buckley/Voorhees stability method, the Voorhees/Buckley swap method and Kendall's rank correlation, with three data sets comprising test collections and submitted runs from NTCIR. Our experiments show that (Average) Normalised Discounted Cumulative Gain at document cut-off l are the best among the rank-based graded-relevance metrics, provided that l is large. On the other hand, if one requires a recall-based graded-relevance metric that is highly correlated with Average Precision, then Q-measure is the best choice. Moreover, these best graded-relevance metrics are at least as stable and sensitive as Average Precision, and are fairly robust to the choice of gain values.

Original languageEnglish
Pages (from-to)531-548
Number of pages18
JournalInformation Processing and Management
Issue number2
Publication statusPublished - 2007 Mar
Externally publishedYes



  • Cumulative gain
  • Evaluation
  • Graded relevance
  • Q-measure
  • Reliability

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems
  • Library and Information Sciences

Cite this