One click one revisited: Enhancing evaluation based on information units

Tetsuya Sakai, Makoto P. Kato

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

This paper extends the evaluation framework of the NTCIR-9 One Click Access Task (1CLICK-1), which required systems to return a single, concise textual output in response to a query in order to satisfy the user immediately after a click on the SEARCH button. Unlike traditional nugget-based summarisation and question answering evaluation methods, S-measure, the official evaluation metric of 1CLICK-1, discounts the value of each information unit based on its position within the textual output. We first show that the discount parameter L of S-measure affects system ranking and discriminative power, and that using multiple values, e.g. L = 250 (user has only 30 seconds to view the text) and L = 500 (user has one minute), is beneficial. We then complement the recall-like S-measure with a simple, precision-like metric called T-measure as well as a combination of S-measure and T-measure, called S#. We show that S# with a heavy emphasis on S-measure imposes an appropriate length penalty to 1CLICK-1 system outputs and yet achieves discriminative power that is comparable to S-measure. These new metrics will be used at NTCIR-10 1CLICK-2.

Original languageEnglish
Title of host publicationInformation Retrieval Technology - 8th Asia Information Retrieval Societies Conference, AIRS 2012, Proceedings
Pages39-51
Number of pages13
DOIs
Publication statusPublished - 2012 Dec 31
Externally publishedYes
Event8th Asia Information Retrieval Societies Conference, AIRS 2012 - Tianjin, China
Duration: 2012 Dec 172012 Dec 19

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7675 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference8th Asia Information Retrieval Societies Conference, AIRS 2012
CountryChina
CityTianjin
Period12/12/1712/12/19

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'One click one revisited: Enhancing evaluation based on information units'. Together they form a unique fingerprint.

  • Cite this

    Sakai, T., & Kato, M. P. (2012). One click one revisited: Enhancing evaluation based on information units. In Information Retrieval Technology - 8th Asia Information Retrieval Societies Conference, AIRS 2012, Proceedings (pp. 39-51). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7675 LNCS). https://doi.org/10.1007/978-3-642-35341-3_4