TY - JOUR
T1 - Ranking the NTCIR systems based on multigrade relevance
AU - Sakai, Tetsuya
PY - 2005
Y1 - 2005
N2 - At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.
AB - At NTCIR-4, new retrieval effectiveness metrics called Q-measure and R-measure were proposed for evaluation based on multigrade relevance. This paper shows that Q-measure inherits both the reliability of noninterpolated Average Precision and the multigrade relevance capability of Average Weighted Precision through a theoretical analysis, and then verify the above claim through experiments by actually ranking the systems submitted to the NTCIR-3 CLIR Task. Our experiments confirm that the Q-measure ranking is very highly correlated with the Average Precision ranking and that it is more reliable than Average Weighted Precision.
UR - http://www.scopus.com/inward/record.url?scp=24344471313&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=24344471313&partnerID=8YFLogxK
U2 - 10.1007/978-3-540-31871-2_22
DO - 10.1007/978-3-540-31871-2_22
M3 - Conference article
AN - SCOPUS:24344471313
VL - 3411
SP - 251
EP - 262
JO - Lecture Notes in Computer Science
JF - Lecture Notes in Computer Science
SN - 0302-9743
T2 - Asia Information Retrieval Symposium, AIRS 2004
Y2 - 18 October 2004 through 20 October 2004
ER -