A Fast and Exact Randomisation Test for Comparing Two Systems with Paired Data

Rikiya Suzuki, Tetsuya Sakai

研究成果: Conference contribution

抄録

The randomisation test with B trials has been used in the information retrieval (IR) field for comparing two systems with paired data (i.e., a common set of topics). It approximates the exact randomisation test whose time complexity is O(2n) for n topics. In this paper, we first show that the counting operation for obtaining the exact p-value in a randomisation test can be reduced to a subsequence sum enumeration problem that can be solved by dynamic programming. By taking advantage of this observation along with the fact that we only require a small number of significant digits in IR evaluation measure scores, we demonstrate that the time complexity of the exact randomisation test can be reduced to O(mn), where m is the maximum subsequence sum of the paired score differences. Hence, researchers can utilise our test if they want to avoid random sampling and/or normality assumptions and want a fast and reliable test. In addition, we utilise Vandermonde's convolution in order to theoretically explain a known fact, namely, that the sign test is a special case of the exact randomisation test.

本文言語English
ホスト出版物のタイトルICTIR 2021 - Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval
出版社Association for Computing Machinery, Inc
ページ239-243
ページ数5
ISBN(電子版)9781450386111
DOI
出版ステータスPublished - 2021 7 11
イベント11th ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2021 - Virtual, Online, Canada
継続期間: 2021 7 11 → …

出版物シリーズ

名前ICTIR 2021 - Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

Conference

Conference11th ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2021
国/地域Canada
CityVirtual, Online
Period21/7/11 → …

ASJC Scopus subject areas

  • コンピュータ サイエンス(その他)
  • 情報システム

フィンガープリント

「A Fast and Exact Randomisation Test for Comparing Two Systems with Paired Data」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル