The reliability of metrics based on graded relevance

Tetsuya Sakai*

*この研究の対応する著者

研究成果: Conference contribution

4 被引用数 (Scopus)

抄録

This paper compares 14 metrics designed for information retrieval evaluation with graded relevance, together with 10 traditional metrics based on binary relevance, in terms of reliability and resemblance of system rankings. More specifically, we use two test collections with submitted runs from the Chinese IR and English IR tasks in the NTCIR-3 CLIR track to examine the metrics using methods proposed by Buckley/Voorhees and Voorhees/Buckley as well as Kendall's rank correlation. Our results show that AnDCGl and nDCGl ((Average) Normalised Discounted Cumulative Gain at Document cut-off l) are good metrics, provided that l is large. However, if one wants to avoid the parameter l altogether, or if one requires a metric that closely resembles TREC Average Precision, then Q-measure appears to be the best choice.

本文言語English
ホスト出版物のタイトルLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
ページ1-16
ページ数16
出版ステータスPublished - 2005 12 1
外部発表はい
イベント2nd Asia Information Retrieval Symposium, AIRS 2005 - Jeju Island, Korea, Republic of
継続期間: 2005 10 132005 10 15

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
3689 LNCS
ISSN(印刷版)0302-9743
ISSN(電子版)1611-3349

Conference

Conference2nd Asia Information Retrieval Symposium, AIRS 2005
国/地域Korea, Republic of
CityJeju Island
Period05/10/1305/10/15

ASJC Scopus subject areas

  • 理論的コンピュータサイエンス
  • コンピュータ サイエンス(全般)

フィンガープリント

「The reliability of metrics based on graded relevance」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル