Simple and effective approach to score standardisation

Tetsuya Sakai*

*この研究の対応する著者

研究成果: Conference contribution

10 被引用数 (Scopus)

抄録

Webber, Moffat and Zobel proposed score standardization for information retrieval evaluation with multiple test collections. Given a topic-by-run raw score matrix in terms of some evaluation measure, each score can be standardised using the topic's sample mean and sample standard deviation across a set of past runs so as to quantify how different a system is from the "average" system in standard deviation units. Using standardised scores, researchers can compare systems across different test collections without worrying about topic hardness or normalisation. While Webber et al. mapped the standardised scores to the [0,1] range using a standard normal cumulative density function, the present study demonstrates that linear transformation of the standardised scores, a method widely used in educational research, can be a simple and effective alternative. We use three TREC robust track data sets with graded relevance assessments and official runs to compare these methods by means of leave-one-out tests, discriminative power, swap rate tests, and topic set size design. In particular, we demonstrate that our method is superior to the method of Webber et al. in terms of swap rates and topic set size design: put simply, our method ensures pairwise system comparisons that are more consistent across different data sets, and is arguably more convenient for designing a new test collection from a statistical viewpoint.

本文言語English
ホスト出版物のタイトルICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval
出版社Association for Computing Machinery, Inc
ページ95-104
ページ数10
ISBN(電子版)9781450344975
DOI
出版ステータスPublished - 2016 9 12
イベント2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016 - Newark, United States
継続期間: 2016 9 122016 9 16

出版物シリーズ

名前ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval

Other

Other2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016
国/地域United States
CityNewark
Period16/9/1216/9/16

ASJC Scopus subject areas

  • 情報システム
  • コンピュータ サイエンス(その他)

フィンガープリント

「Simple and effective approach to score standardisation」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル