Ranking the NTCIR systems based on multigrade relevance

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science
EditorsS.H. Myaeng, M. Zhou, H.J. Zhang, K.-F. Wong
Pages251-262
Number of pages12
Volume3411
Publication statusPublished - 2005
Externally publishedYes
EventAsia Information Retrieval Symposium, AIRS 2004 - Beijing, China
Duration: 2004 Oct 182004 Oct 20

Other

OtherAsia Information Retrieval Symposium, AIRS 2004
CountryChina
CityBeijing
Period04/10/1804/10/20

Fingerprint

Experiments

ASJC Scopus subject areas

  • Computer Science (miscellaneous)

Cite this

Sakai, T. (2005). Ranking the NTCIR systems based on multigrade relevance. In S. H. Myaeng, M. Zhou, H. J. Zhang, & K-F. Wong (Eds.), Lecture Notes in Computer Science (Vol. 3411, pp. 251-262)

Ranking the NTCIR systems based on multigrade relevance. / Sakai, Tetsuya.

Lecture Notes in Computer Science. ed. / S.H. Myaeng; M. Zhou; H.J. Zhang; K.-F. Wong. Vol. 3411 2005. p. 251-262.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sakai, T 2005, Ranking the NTCIR systems based on multigrade relevance. in SH Myaeng, M Zhou, HJ Zhang & K-F Wong (eds), Lecture Notes in Computer Science. vol. 3411, pp. 251-262, Asia Information Retrieval Symposium, AIRS 2004, Beijing, China, 04/10/18.
Sakai T. Ranking the NTCIR systems based on multigrade relevance. In Myaeng SH, Zhou M, Zhang HJ, Wong K-F, editors, Lecture Notes in Computer Science. Vol. 3411. 2005. p. 251-262
Sakai, Tetsuya. / Ranking the NTCIR systems based on multigrade relevance. Lecture Notes in Computer Science. editor / S.H. Myaeng ; M. Zhou ; H.J. Zhang ; K.-F. Wong. Vol. 3411 2005. pp. 251-262
@inproceedings{e3a29070688347bd890deca27c3d764a,
title = "Ranking the NTCIR systems based on multigrade relevance",
abstract = "At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.",
author = "Tetsuya Sakai",
year = "2005",
language = "English",
volume = "3411",
pages = "251--262",
editor = "S.H. Myaeng and M. Zhou and H.J. Zhang and K.-F. Wong",
booktitle = "Lecture Notes in Computer Science",

}

TY - GEN

T1 - Ranking the NTCIR systems based on multigrade relevance

AU - Sakai, Tetsuya

PY - 2005

Y1 - 2005

N2 - At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.

AB - At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.

UR - http://www.scopus.com/inward/record.url?scp=24344471313&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=24344471313&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:24344471313

VL - 3411

SP - 251

EP - 262

BT - Lecture Notes in Computer Science

A2 - Myaeng, S.H.

A2 - Zhou, M.

A2 - Zhang, H.J.

A2 - Wong, K.-F.

ER -