Report from the NTCIR-10 1CLICK-2 Japanese subtask: Baselines, upperbounds and evaluation robustness

Makoto P. Kato, Tetsuya Sakai, Takehiro Yamamoto, Mayu Iwata

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

The One Click Access Task (1CLICK) of NTCIR requires systems to return a concise multi-document summary of web pages in response to a query which is assumed to have been submitted in a mobile context. Systems are evaluated based on information units (or iUnits), and are required to present important pieces of information first and to minimise the amount of text the user has to read. Using the official Japanese results of the second round of the 1CLICK task from NTCIR-10, we discuss our task setting and evaluation framework. Our analyses show that: (1) Simple baseline methods that leverage search engine snippets or Wikipedia are effective for "lookup" type queries but not necessarily for other query types; (2) There is still a substantial gap between manual and automatic runs; and (3) Our evaluation metrics are relatively robust to the incompleteness of iUnits.

Original languageEnglish
Title of host publicationSIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval
Pages753-756
Number of pages4
DOIs
Publication statusPublished - 2013
Externally publishedYes
Event36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2013 - Dublin
Duration: 2013 Jul 282013 Aug 1

Other

Other36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2013
CityDublin
Period13/7/2813/8/1

Fingerprint

Search engines
Websites

Keywords

  • Evaluation
  • Information units
  • Mobile environment
  • NTCIR
  • Nuggets
  • Summaries
  • Test collections

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Information Systems

Cite this

Kato, M. P., Sakai, T., Yamamoto, T., & Iwata, M. (2013). Report from the NTCIR-10 1CLICK-2 Japanese subtask: Baselines, upperbounds and evaluation robustness. In SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 753-756) https://doi.org/10.1145/2484028.2484117

Report from the NTCIR-10 1CLICK-2 Japanese subtask : Baselines, upperbounds and evaluation robustness. / Kato, Makoto P.; Sakai, Tetsuya; Yamamoto, Takehiro; Iwata, Mayu.

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2013. p. 753-756.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kato, MP, Sakai, T, Yamamoto, T & Iwata, M 2013, Report from the NTCIR-10 1CLICK-2 Japanese subtask: Baselines, upperbounds and evaluation robustness. in SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 753-756, 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2013, Dublin, 13/7/28. https://doi.org/10.1145/2484028.2484117
Kato MP, Sakai T, Yamamoto T, Iwata M. Report from the NTCIR-10 1CLICK-2 Japanese subtask: Baselines, upperbounds and evaluation robustness. In SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2013. p. 753-756 https://doi.org/10.1145/2484028.2484117
Kato, Makoto P. ; Sakai, Tetsuya ; Yamamoto, Takehiro ; Iwata, Mayu. / Report from the NTCIR-10 1CLICK-2 Japanese subtask : Baselines, upperbounds and evaluation robustness. SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2013. pp. 753-756
@inproceedings{dc1b6da1a0894d3da69562c664bccacb,
title = "Report from the NTCIR-10 1CLICK-2 Japanese subtask: Baselines, upperbounds and evaluation robustness",
abstract = "The One Click Access Task (1CLICK) of NTCIR requires systems to return a concise multi-document summary of web pages in response to a query which is assumed to have been submitted in a mobile context. Systems are evaluated based on information units (or iUnits), and are required to present important pieces of information first and to minimise the amount of text the user has to read. Using the official Japanese results of the second round of the 1CLICK task from NTCIR-10, we discuss our task setting and evaluation framework. Our analyses show that: (1) Simple baseline methods that leverage search engine snippets or Wikipedia are effective for {"}lookup{"} type queries but not necessarily for other query types; (2) There is still a substantial gap between manual and automatic runs; and (3) Our evaluation metrics are relatively robust to the incompleteness of iUnits.",
keywords = "Evaluation, Information units, Mobile environment, NTCIR, Nuggets, Summaries, Test collections",
author = "Kato, {Makoto P.} and Tetsuya Sakai and Takehiro Yamamoto and Mayu Iwata",
year = "2013",
doi = "10.1145/2484028.2484117",
language = "English",
isbn = "9781450320344",
pages = "753--756",
booktitle = "SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval",

}

TY - GEN

T1 - Report from the NTCIR-10 1CLICK-2 Japanese subtask

T2 - Baselines, upperbounds and evaluation robustness

AU - Kato, Makoto P.

AU - Sakai, Tetsuya

AU - Yamamoto, Takehiro

AU - Iwata, Mayu

PY - 2013

Y1 - 2013

N2 - The One Click Access Task (1CLICK) of NTCIR requires systems to return a concise multi-document summary of web pages in response to a query which is assumed to have been submitted in a mobile context. Systems are evaluated based on information units (or iUnits), and are required to present important pieces of information first and to minimise the amount of text the user has to read. Using the official Japanese results of the second round of the 1CLICK task from NTCIR-10, we discuss our task setting and evaluation framework. Our analyses show that: (1) Simple baseline methods that leverage search engine snippets or Wikipedia are effective for "lookup" type queries but not necessarily for other query types; (2) There is still a substantial gap between manual and automatic runs; and (3) Our evaluation metrics are relatively robust to the incompleteness of iUnits.

AB - The One Click Access Task (1CLICK) of NTCIR requires systems to return a concise multi-document summary of web pages in response to a query which is assumed to have been submitted in a mobile context. Systems are evaluated based on information units (or iUnits), and are required to present important pieces of information first and to minimise the amount of text the user has to read. Using the official Japanese results of the second round of the 1CLICK task from NTCIR-10, we discuss our task setting and evaluation framework. Our analyses show that: (1) Simple baseline methods that leverage search engine snippets or Wikipedia are effective for "lookup" type queries but not necessarily for other query types; (2) There is still a substantial gap between manual and automatic runs; and (3) Our evaluation metrics are relatively robust to the incompleteness of iUnits.

KW - Evaluation

KW - Information units

KW - Mobile environment

KW - NTCIR

KW - Nuggets

KW - Summaries

KW - Test collections

UR - http://www.scopus.com/inward/record.url?scp=84883070185&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84883070185&partnerID=8YFLogxK

U2 - 10.1145/2484028.2484117

DO - 10.1145/2484028.2484117

M3 - Conference contribution

AN - SCOPUS:84883070185

SN - 9781450320344

SP - 753

EP - 756

BT - SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

ER -