The effect of back-formulating questions in question answering evaluation

Tetsuya Sakai, Yoshimi Saito, Yumi Ichimura, Tomoharu Kokubu, Makoto Koyama

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The effect of back-formulating questions in question answering evaluation is discussed. QA evaluation using back-formulated questions is not equivalent to that using questions devised without access to the document collection. QA test collection constructors should be careful when back-formulating questions based on some document passages. The users of back-formulated QA test collections should bear in mind that the absolute QA performance values obtained through experiments may not represent operational QA performance.

Original languageEnglish
Title of host publicationProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
EditorsK. Jarvelin, J. Allen, P. Bruza, M. Sanderson
Pages474-475
Number of pages2
Publication statusPublished - 2004
Externally publishedYes
EventProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - Sheffield
Duration: 2004 Jul 252004 Jul 29

Other

OtherProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
CitySheffield
Period04/7/2504/7/29

Fingerprint

Experiments

Keywords

  • Evaluation
  • Question answering
  • Test collection

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Sakai, T., Saito, Y., Ichimura, Y., Kokubu, T., & Koyama, M. (2004). The effect of back-formulating questions in question answering evaluation. In K. Jarvelin, J. Allen, P. Bruza, & M. Sanderson (Eds.), Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 474-475)

The effect of back-formulating questions in question answering evaluation. / Sakai, Tetsuya; Saito, Yoshimi; Ichimura, Yumi; Kokubu, Tomoharu; Koyama, Makoto.

Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ed. / K. Jarvelin; J. Allen; P. Bruza; M. Sanderson. 2004. p. 474-475.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sakai, T, Saito, Y, Ichimura, Y, Kokubu, T & Koyama, M 2004, The effect of back-formulating questions in question answering evaluation. in K Jarvelin, J Allen, P Bruza & M Sanderson (eds), Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 474-475, Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, 04/7/25.
Sakai T, Saito Y, Ichimura Y, Kokubu T, Koyama M. The effect of back-formulating questions in question answering evaluation. In Jarvelin K, Allen J, Bruza P, Sanderson M, editors, Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2004. p. 474-475
Sakai, Tetsuya ; Saito, Yoshimi ; Ichimura, Yumi ; Kokubu, Tomoharu ; Koyama, Makoto. / The effect of back-formulating questions in question answering evaluation. Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. editor / K. Jarvelin ; J. Allen ; P. Bruza ; M. Sanderson. 2004. pp. 474-475
@inproceedings{588385a457ef4ff9ab31063b46c98b6b,
title = "The effect of back-formulating questions in question answering evaluation",
abstract = "The effect of back-formulating questions in question answering evaluation is discussed. QA evaluation using back-formulated questions is not equivalent to that using questions devised without access to the document collection. QA test collection constructors should be careful when back-formulating questions based on some document passages. The users of back-formulated QA test collections should bear in mind that the absolute QA performance values obtained through experiments may not represent operational QA performance.",
keywords = "Evaluation, Question answering, Test collection",
author = "Tetsuya Sakai and Yoshimi Saito and Yumi Ichimura and Tomoharu Kokubu and Makoto Koyama",
year = "2004",
language = "English",
isbn = "1581138814",
pages = "474--475",
editor = "K. Jarvelin and J. Allen and P. Bruza and M. Sanderson",
booktitle = "Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval",

}

TY - GEN

T1 - The effect of back-formulating questions in question answering evaluation

AU - Sakai, Tetsuya

AU - Saito, Yoshimi

AU - Ichimura, Yumi

AU - Kokubu, Tomoharu

AU - Koyama, Makoto

PY - 2004

Y1 - 2004

N2 - The effect of back-formulating questions in question answering evaluation is discussed. QA evaluation using back-formulated questions is not equivalent to that using questions devised without access to the document collection. QA test collection constructors should be careful when back-formulating questions based on some document passages. The users of back-formulated QA test collections should bear in mind that the absolute QA performance values obtained through experiments may not represent operational QA performance.

AB - The effect of back-formulating questions in question answering evaluation is discussed. QA evaluation using back-formulated questions is not equivalent to that using questions devised without access to the document collection. QA test collection constructors should be careful when back-formulating questions based on some document passages. The users of back-formulated QA test collections should bear in mind that the absolute QA performance values obtained through experiments may not represent operational QA performance.

KW - Evaluation

KW - Question answering

KW - Test collection

UR - http://www.scopus.com/inward/record.url?scp=8644283799&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=8644283799&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:8644283799

SN - 1581138814

SP - 474

EP - 475

BT - Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

A2 - Jarvelin, K.

A2 - Allen, J.

A2 - Bruza, P.

A2 - Sanderson, M.

ER -