On the Instability of Diminishing Return IR Measures

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The diminishing return property of ERR (Expected Reciprocal Rank) is highly intuitive and attractive: its user model says, for example, that after the users have found a highly relevant document at rank r, few of them will continue to examine rank (r+ 1 ) and beyond. Recently, another IR evaluation measure based on diminishing return called iRBU (intentwise Rank-Biased Utility) was proposed, and it was reported that nDCG (normalised Discounted Cumulative Gain) and iRBU align surprisingly well with users’ SERP (Search Engine Result Page) preferences. The present study conducts offline evaluations of diminishing return measures including ERR and iRBU along with other popular measures such as nDCG, using four test collections and the associated runs from recent TREC tracks and NTCIR tasks. Our results show that the diminishing return measures generally underperform other graded relevance measures in terms of system ranking consistency across two disjoint topic sets as well as discriminative power. The results generalise a previous finding on ERR regarding its limited discriminative power, showing that the diminishing return user model hurts the stability of evaluation measures regardless of the utility function part of the measure. Hence, while we do recommend iRBU along with nDCG for evaluating adhoc IR systems from multiple user-oriented angles, iRBU should be used under the awareness that it can be much less statistically stable than nDCG.

Original languageEnglish
Title of host publicationAdvances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings
EditorsDjoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, Fabrizio Sebastiani
PublisherSpringer Science and Business Media Deutschland GmbH
Pages572-586
Number of pages15
ISBN (Print)9783030721121
DOIs
Publication statusPublished - 2021
Event43rd European Conference on Information Retrieval Research, ECIR 2021 - Virtual, Online
Duration: 2021 Mar 282021 Apr 1

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12656 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference43rd European Conference on Information Retrieval Research, ECIR 2021
CityVirtual, Online
Period21/3/2821/4/1

Keywords

  • Diminishing return
  • Discriminative power
  • Evaluation measures
  • Statistical significance
  • System ranking consistency

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'On the Instability of Diminishing Return IR Measures'. Together they form a unique fingerprint.

Cite this