TY - GEN
T1 - Exploring semi-automatic nugget extraction for Japanese one click access evaluation
AU - Ekstrand-Abueg, Matthew
AU - Pavlu, Virgil
AU - Kato, Makoto P.
AU - Sakai, Tetsuya
AU - Yamamoto, Takehiro
AU - Iwata, Mayu
PY - 2013
Y1 - 2013
N2 - Building test collections based on nuggets is useful evaluating systems that return documents, answers, or summaries. However, nugget construction requires a lot of manual work and is not feasible for large query sets. Towards an efficient and scalable nugget-based evaluation, we study the applicability of semi-automatic nugget extraction in the context of the ongoing NTCIR One Click Access (1CLICK) task. We compare manually-extracted and semi-automatically- extracted Japanese nuggets to demonstrate the coverage and efficiency of the semi-automatic nugget extraction. Our findings suggest that the manual nugget extraction can be replaced with a direct adaptation of the English semi-automatic nugget extraction system, especially for queries for which the user desires broad answers from free-form text.
AB - Building test collections based on nuggets is useful evaluating systems that return documents, answers, or summaries. However, nugget construction requires a lot of manual work and is not feasible for large query sets. Towards an efficient and scalable nugget-based evaluation, we study the applicability of semi-automatic nugget extraction in the context of the ongoing NTCIR One Click Access (1CLICK) task. We compare manually-extracted and semi-automatically- extracted Japanese nuggets to demonstrate the coverage and efficiency of the semi-automatic nugget extraction. Our findings suggest that the manual nugget extraction can be replaced with a direct adaptation of the English semi-automatic nugget extraction system, especially for queries for which the user desires broad answers from free-form text.
KW - Evaluation
KW - Information units
KW - NTCIR
KW - Nuggets
KW - Summaries
KW - Test collections
UR - http://www.scopus.com/inward/record.url?scp=84883094511&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84883094511&partnerID=8YFLogxK
U2 - 10.1145/2484028.2484153
DO - 10.1145/2484028.2484153
M3 - Conference contribution
AN - SCOPUS:84883094511
SN - 9781450320344
T3 - SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
SP - 749
EP - 752
BT - SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
T2 - 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2013
Y2 - 28 July 2013 through 1 August 2013
ER -