Generalising Kendall's Tau for noisy and incomplete preference judgements

Riku Togashi, Tetsuya Sakai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We propose a new ranking evaluation measure for situations where multiple preference judgements are given for each item pair but they may be noisy (i.e., some judgements are unreliable) and/or incomplete (i.e., some judgements are missing). While it is generally easier for assessors to conduct preference judgements than absolute judgements, it is often not practical to obtain preference judgements for all combinations of documents. However, this problem can be overcome if we can effectively utilise noisy and incomplete preference judgements such as those that can be obtained from crowdsourcing. Our measure, η, is based on a simple probabilistic user model of the labellers which assumes that each document is associated with a graded relevance score for a given query. We also consider situations where multiple preference probabilities, rather than preference labels, are given for each document pair. For example, in the absence of manual preference judgements, one might want to employ an ensemble of machine learning techniques to obtain such estimated probabilities. For this scenario, we propose another ranking evaluation measure called ηp . Through simulated experiments, we demonstrate that our proposed measures τ and ηp can evaluate rankings more reliably than τ -b, a popular rank correlation measure.

Original languageEnglish
Title of host publicationICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages193-196
Number of pages4
ISBN (Electronic)9781450368810
DOIs
Publication statusPublished - 2019 Sep 23
Event9th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2019 - Santa Clara, United States
Duration: 2019 Oct 22019 Oct 5

Publication series

NameICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

Conference

Conference9th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2019
CountryUnited States
CitySanta Clara
Period19/10/219/10/5

    Fingerprint

Keywords

  • Crowdsourcing
  • Evaluation measures
  • Graded relevance
  • Preference judgements

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Information Systems

Cite this

Togashi, R., & Sakai, T. (2019). Generalising Kendall's Tau for noisy and incomplete preference judgements. In ICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (pp. 193-196). (ICTIR 2019 - Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval). Association for Computing Machinery, Inc. https://doi.org/10.1145/3341981.3344246