Click the search button and be happy: Evaluating direct and immediate information access

Tetsuya Sakai, Makoto P. Kato, Young In Song

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

We define Direct Information Access as a type of information access where there is no user operation such as clicking or scrolling between the user's click on the search button and the user's information acquisition; we define Immediate Information Access as a type of information access where the user can locate the relevant information within the system output very quickly. Hence, a Direct and Immediate Information Access (DIIA) system is expected to satisfy the user's information need very quickly with its very first response. We propose a nugget-based evaluation framework for DIIA, which takes nugget positions into account in order to evaluate the ability of a system to present important nuggets first and to minimise the amount of text the user has to read. To demonstrate the integrity, usefulness and limitations of our framework, we built a Japanese DIIA test collection with 60 queries and over 2,800 nuggets as well as an offset-based nugget match evaluation interface, and conducted experiments with manual and automatic runs. The results suggest our proposal is a useful complement to traditional ranked retrieval evaluation based on document relevance.

Original languageEnglish
Title of host publicationInternational Conference on Information and Knowledge Management, Proceedings
Pages621-630
Number of pages10
DOIs
Publication statusPublished - 2011
Externally publishedYes
Event20th ACM Conference on Information and Knowledge Management, CIKM'11 - Glasgow
Duration: 2011 Oct 242011 Oct 28

Other

Other20th ACM Conference on Information and Knowledge Management, CIKM'11
CityGlasgow
Period11/10/2411/10/28

Fingerprint

Evaluation
Information needs
Information acquisition
Usefulness
Test collections
Experiment
Integrity
Query

Keywords

  • evaluation
  • information access
  • nugget
  • test collection

ASJC Scopus subject areas

  • Business, Management and Accounting(all)
  • Decision Sciences(all)

Cite this

Sakai, T., Kato, M. P., & Song, Y. I. (2011). Click the search button and be happy: Evaluating direct and immediate information access. In International Conference on Information and Knowledge Management, Proceedings (pp. 621-630) https://doi.org/10.1145/2063576.2063669

Click the search button and be happy : Evaluating direct and immediate information access. / Sakai, Tetsuya; Kato, Makoto P.; Song, Young In.

International Conference on Information and Knowledge Management, Proceedings. 2011. p. 621-630.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sakai, T, Kato, MP & Song, YI 2011, Click the search button and be happy: Evaluating direct and immediate information access. in International Conference on Information and Knowledge Management, Proceedings. pp. 621-630, 20th ACM Conference on Information and Knowledge Management, CIKM'11, Glasgow, 11/10/24. https://doi.org/10.1145/2063576.2063669
Sakai T, Kato MP, Song YI. Click the search button and be happy: Evaluating direct and immediate information access. In International Conference on Information and Knowledge Management, Proceedings. 2011. p. 621-630 https://doi.org/10.1145/2063576.2063669
Sakai, Tetsuya ; Kato, Makoto P. ; Song, Young In. / Click the search button and be happy : Evaluating direct and immediate information access. International Conference on Information and Knowledge Management, Proceedings. 2011. pp. 621-630
@inproceedings{39aa947942844e4a8fef024ff6572c63,
title = "Click the search button and be happy: Evaluating direct and immediate information access",
abstract = "We define Direct Information Access as a type of information access where there is no user operation such as clicking or scrolling between the user's click on the search button and the user's information acquisition; we define Immediate Information Access as a type of information access where the user can locate the relevant information within the system output very quickly. Hence, a Direct and Immediate Information Access (DIIA) system is expected to satisfy the user's information need very quickly with its very first response. We propose a nugget-based evaluation framework for DIIA, which takes nugget positions into account in order to evaluate the ability of a system to present important nuggets first and to minimise the amount of text the user has to read. To demonstrate the integrity, usefulness and limitations of our framework, we built a Japanese DIIA test collection with 60 queries and over 2,800 nuggets as well as an offset-based nugget match evaluation interface, and conducted experiments with manual and automatic runs. The results suggest our proposal is a useful complement to traditional ranked retrieval evaluation based on document relevance.",
keywords = "evaluation, information access, nugget, test collection",
author = "Tetsuya Sakai and Kato, {Makoto P.} and Song, {Young In}",
year = "2011",
doi = "10.1145/2063576.2063669",
language = "English",
isbn = "9781450307178",
pages = "621--630",
booktitle = "International Conference on Information and Knowledge Management, Proceedings",

}

TY - GEN

T1 - Click the search button and be happy

T2 - Evaluating direct and immediate information access

AU - Sakai, Tetsuya

AU - Kato, Makoto P.

AU - Song, Young In

PY - 2011

Y1 - 2011

N2 - We define Direct Information Access as a type of information access where there is no user operation such as clicking or scrolling between the user's click on the search button and the user's information acquisition; we define Immediate Information Access as a type of information access where the user can locate the relevant information within the system output very quickly. Hence, a Direct and Immediate Information Access (DIIA) system is expected to satisfy the user's information need very quickly with its very first response. We propose a nugget-based evaluation framework for DIIA, which takes nugget positions into account in order to evaluate the ability of a system to present important nuggets first and to minimise the amount of text the user has to read. To demonstrate the integrity, usefulness and limitations of our framework, we built a Japanese DIIA test collection with 60 queries and over 2,800 nuggets as well as an offset-based nugget match evaluation interface, and conducted experiments with manual and automatic runs. The results suggest our proposal is a useful complement to traditional ranked retrieval evaluation based on document relevance.

AB - We define Direct Information Access as a type of information access where there is no user operation such as clicking or scrolling between the user's click on the search button and the user's information acquisition; we define Immediate Information Access as a type of information access where the user can locate the relevant information within the system output very quickly. Hence, a Direct and Immediate Information Access (DIIA) system is expected to satisfy the user's information need very quickly with its very first response. We propose a nugget-based evaluation framework for DIIA, which takes nugget positions into account in order to evaluate the ability of a system to present important nuggets first and to minimise the amount of text the user has to read. To demonstrate the integrity, usefulness and limitations of our framework, we built a Japanese DIIA test collection with 60 queries and over 2,800 nuggets as well as an offset-based nugget match evaluation interface, and conducted experiments with manual and automatic runs. The results suggest our proposal is a useful complement to traditional ranked retrieval evaluation based on document relevance.

KW - evaluation

KW - information access

KW - nugget

KW - test collection

UR - http://www.scopus.com/inward/record.url?scp=83055168077&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=83055168077&partnerID=8YFLogxK

U2 - 10.1145/2063576.2063669

DO - 10.1145/2063576.2063669

M3 - Conference contribution

AN - SCOPUS:83055168077

SN - 9781450307178

SP - 621

EP - 630

BT - International Conference on Information and Knowledge Management, Proceedings

ER -