Ranking the NTCIR systems based on multigrade relevance

Tetsuya Sakai

doi:10.1007/978-3-540-31871-2_22

Ranking the NTCIR systems based on multigrade relevance

Tetsuya Sakai^*

^*Corresponding author for this work

Research output: Contribution to journal › Conference article › peer-review

11 Citations (Scopus)

Abstract

At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.

Original language	English
Pages (from-to)	251-262
Number of pages	12
Journal	LECTURE NOTES IN COMPUTER SCIENCE
Volume	3411
DOIs	https://doi.org/10.1007/978-3-540-31871-2_22
Publication status	Published - 2005
Externally published	Yes
Event	Asia Information Retrieval Symposium, AIRS 2004 - Beijing, China Duration: 2004 Oct 18 → 2004 Oct 20

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-540-31871-2_22

Cite this

@article{e3a29070688347bd890deca27c3d764a,

title = "Ranking the NTCIR systems based on multigrade relevance",

abstract = "At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.",

author = "Tetsuya Sakai",

year = "2005",

doi = "10.1007/978-3-540-31871-2_22",

language = "English",

volume = "3411",

pages = "251--262",

journal = "Lecture Notes in Computer Science",

issn = "0302-9743",

publisher = "Springer Verlag",

note = "Asia Information Retrieval Symposium, AIRS 2004 ; Conference date: 18-10-2004 Through 20-10-2004",

}

TY - JOUR

T1 - Ranking the NTCIR systems based on multigrade relevance

AU - Sakai, Tetsuya

PY - 2005

Y1 - 2005

N2 - At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.

AB - At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.

UR - http://www.scopus.com/inward/record.url?scp=24344471313&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=24344471313&partnerID=8YFLogxK

U2 - 10.1007/978-3-540-31871-2_22

DO - 10.1007/978-3-540-31871-2_22

M3 - Conference article

AN - SCOPUS:24344471313

SN - 0302-9743

VL - 3411

SP - 251

EP - 262

JO - Lecture Notes in Computer Science

JF - Lecture Notes in Computer Science

T2 - Asia Information Retrieval Symposium, AIRS 2004

Y2 - 18 October 2004 through 20 October 2004

ER -

Ranking the NTCIR systems based on multigrade relevance

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this