Generalising Kendall's Tau for noisy and incomplete preference judgements

Riku Togashi, Tetsuya Sakai

研究成果: Conference contribution

抜粋

We propose a new ranking evaluation measure for situations where multiple preference judgements are given for each item pair but they may be noisy (i.e., some judgements are unreliable) and/or incomplete (i.e., some judgements are missing). While it is generally easier for assessors to conduct preference judgements than absolute judgements, it is often not practical to obtain preference judgements for all combinations of documents. However, this problem can be overcome if we can effectively utilise noisy and incomplete preference judgements such as those that can be obtained from crowdsourcing. Our measure, η, is based on a simple probabilistic user model of the labellers which assumes that each document is associated with a graded relevance score for a given query. We also consider situations where multiple preference probabilities, rather than preference labels, are given for each document pair. For example, in the absence of manual preference judgements, one might want to employ an ensemble of machine learning techniques to obtain such estimated probabilities. For this scenario, we propose another ranking evaluation measure called ηp . Through simulated experiments, we demonstrate that our proposed measures τ and ηp can evaluate rankings more reliably than τ -b, a popular rank correlation measure.

元の言語English
ホスト出版物のタイトルICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval
出版者Association for Computing Machinery, Inc
ページ193-196
ページ数4
ISBN(電子版)9781450368810
DOI
出版物ステータスPublished - 2019 9 23
イベント9th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2019 - Santa Clara, United States
継続期間: 2019 10 22019 10 5

出版物シリーズ

名前ICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

Conference

Conference9th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2019
United States
Santa Clara
期間19/10/219/10/5

    フィンガープリント

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Information Systems

これを引用

Togashi, R., & Sakai, T. (2019). Generalising Kendall's Tau for noisy and incomplete preference judgements. : ICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (pp. 193-196). (ICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval). Association for Computing Machinery, Inc. https://doi.org/10.1145/3341981.3344246