The reliability of metrics based on graded relevance

研究成果: Conference contribution

3 引用 (Scopus)

抜粋

This paper compares 14 metrics designed for information retrieval evaluation with graded relevance, together with 10 traditional metrics based on binary relevance, in terms of reliability and resemblance of system rankings. More specifically, we use two test collections with submitted runs from the Chinese IR and English IR tasks in the NTCIR-3 CLIR track to examine the metrics using methods proposed by Buckley/Voorhees and Voorhees/Buckley as well as Kendall's rank correlation. Our results show that AnDCGl and nDCGl ((Average) Normalised Discounted Cumulative Gain at Document cut-off l) are good metrics, provided that l is large. However, if one wants to avoid the parameter l altogether, or if one requires a metric that closely resembles TREC Average Precision, then Q-measure appears to be the best choice.

元の言語English
ホスト出版物のタイトルLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
ページ1-16
ページ数16
出版物ステータスPublished - 2005 12 1
イベント2nd Asia Information Retrieval Symposium, AIRS 2005 - Jeju Island, Korea, Republic of
継続期間: 2005 10 132005 10 15

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
3689 LNCS
ISSN(印刷物)0302-9743
ISSN(電子版)1611-3349

Conference

Conference2nd Asia Information Retrieval Symposium, AIRS 2005
Korea, Republic of
Jeju Island
期間05/10/1305/10/15

    フィンガープリント

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

これを引用

Sakai, T. (2005). The reliability of metrics based on graded relevance. : Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 1-16). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 巻数 3689 LNCS).