On the Instability of Diminishing Return IR Measures

研究成果: Conference contribution

抄録

The diminishing return property of ERR (Expected Reciprocal Rank) is highly intuitive and attractive: its user model says, for example, that after the users have found a highly relevant document at rank r, few of them will continue to examine rank (r+ 1 ) and beyond. Recently, another IR evaluation measure based on diminishing return called iRBU (intentwise Rank-Biased Utility) was proposed, and it was reported that nDCG (normalised Discounted Cumulative Gain) and iRBU align surprisingly well with users’ SERP (Search Engine Result Page) preferences. The present study conducts offline evaluations of diminishing return measures including ERR and iRBU along with other popular measures such as nDCG, using four test collections and the associated runs from recent TREC tracks and NTCIR tasks. Our results show that the diminishing return measures generally underperform other graded relevance measures in terms of system ranking consistency across two disjoint topic sets as well as discriminative power. The results generalise a previous finding on ERR regarding its limited discriminative power, showing that the diminishing return user model hurts the stability of evaluation measures regardless of the utility function part of the measure. Hence, while we do recommend iRBU along with nDCG for evaluating adhoc IR systems from multiple user-oriented angles, iRBU should be used under the awareness that it can be much less statistically stable than nDCG.

本文言語English
ホスト出版物のタイトルAdvances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings
編集者Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, Fabrizio Sebastiani
出版社Springer Science and Business Media Deutschland GmbH
ページ572-586
ページ数15
ISBN(印刷版)9783030721121
DOI
出版ステータスPublished - 2021
イベント43rd European Conference on Information Retrieval Research, ECIR 2021 - Virtual, Online
継続期間: 2021 3 282021 4 1

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
12656 LNCS
ISSN(印刷版)0302-9743
ISSN(電子版)1611-3349

Conference

Conference43rd European Conference on Information Retrieval Research, ECIR 2021
CityVirtual, Online
Period21/3/2821/4/1

ASJC Scopus subject areas

  • 理論的コンピュータサイエンス
  • コンピュータ サイエンス(全般)

フィンガープリント

「On the Instability of Diminishing Return IR Measures」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル