Good Evaluation Measures based on Document Preferences

Tetsuya Sakai, Zhaohao Zeng

研究成果: Conference contribution

9 被引用数 (Scopus)

抄録

For offline evaluation of IR systems, some researchers have proposed to utilise pairwise document preference assessments instead of relevance assessments of individual documents, as it may be easier for assessors to make relative decisions rather than absolute ones. Simple preference-based evaluation measures such as ppref and wpref have been proposed, but the past decade did not see any wide use of such measures. One reason for this may be that, while these new measures have been reported to behave more or less similarly to traditional measures based on absolute assessments, whether they actually align with the users' perception of search engine result pages (SERPs) has been unknown. The present study addresses exactly this question, after formally defining two classes of preference-based measures called Pref measures and Î"-measures. We show that the best of these measures perform at least as well as an average assessor in terms of agreement with users' SERP preferences, and that implicit document preferences (i.e., those suggested by a SERP that retrieves one document but not the other) play a much more important role than explicit preferences (i.e., those suggested by a SERP that retrieves one document above the other). We have released our data set containing 119,646 document preferences, so that the feasibility of document preferenced-based evaluation can be further pursued by the IR community.

本文言語English
ホスト出版物のタイトルSIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
出版社Association for Computing Machinery, Inc
ページ359-368
ページ数10
ISBN(電子版)9781450380164
DOI
出版ステータスPublished - 2020 7月 25
イベント43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020 - Virtual, Online, China
継続期間: 2020 7月 252020 7月 30

出版物シリーズ

名前SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Conference

Conference43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020
国/地域China
CityVirtual, Online
Period20/7/2520/7/30

ASJC Scopus subject areas

  • コンピュータ グラフィックスおよびコンピュータ支援設計
  • 情報システム
  • ソフトウェア

フィンガープリント

「Good Evaluation Measures based on Document Preferences」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル