Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model

Nuo Chen, Fan Zhang, Tetsuya Sakai

研究成果: Conference contribution

抄録

Models of existing evaluation metrics assume that users are rational decision-makers trying to pursue maximised utility. However, studies in behavioural economics show that people are not always rational when making decisions. Previous studies showed that the anchoring effect can influence the relevance judgement of a document. In this paper, we challenge the rational user assumption and introduce the anchoring effect into user models. We first propose a framework for query-level evaluation metrics by incorporating the anchoring effect into the user model. In the framework, the magnitude of the anchoring effect is related to the quality of the previous document. We then apply our framework to several query-level evaluation metrics and compare them with their vanilla version as the baseline in terms of user satisfaction on a publicly available search dataset. As a result, our Anchoring-aware Metrics (AMs) outperformed their baselines in term of correlation with user satisfaction. The result suggests that we can better predict user query satisfaction feedbacks by incorporating the anchoring effect into user models of existing evaluating metrics. As far as we know, we are the first to introduce the anchoring effect into information retrieval evaluation metrics. Our findings provide a perspective from behavioural economics to better understand user behaviour and satisfaction in search interaction.

本文言語English
ホスト出版物のタイトルSIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
出版社Association for Computing Machinery, Inc
ページ2709-2714
ページ数6
ISBN(電子版)9781450387323
DOI
出版ステータスPublished - 2022 7月 6
イベント45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022 - Madrid, Spain
継続期間: 2022 7月 112022 7月 15

出版物シリーズ

名前SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Conference

Conference45th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022
国/地域Spain
CityMadrid
Period22/7/1122/7/15

ASJC Scopus subject areas

  • コンピュータ グラフィックスおよびコンピュータ支援設計
  • 情報システム
  • ソフトウェア

フィンガープリント

「Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル