The effect of back-formulating questions in question answering evaluation

Tetsuya Sakai, Yoshimi Saito, Yumi Ichimura, Tomoharu Kokubu, Makoto Koyama

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The effect of back-formulating questions in question answering evaluation is discussed. QA evaluation using back-formulated questions is not equivalent to that using questions devised without access to the document collection. QA test collection constructors should be careful when back-formulating questions based on some document passages. The users of back-formulated QA test collections should bear in mind that the absolute QA performance values obtained through experiments may not represent operational QA performance.

Original languageEnglish
Title of host publicationProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
EditorsK. Jarvelin, J. Allen, P. Bruza, M. Sanderson
Pages474-475
Number of pages2
Publication statusPublished - 2004 Nov 25
EventProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval - Sheffield, United Kingdom
Duration: 2004 Jul 252004 Jul 29

Publication series

NameProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Conference

ConferenceProceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
CountryUnited Kingdom
CitySheffield
Period04/7/2504/7/29

    Fingerprint

Keywords

  • Evaluation
  • Question answering
  • Test collection

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Sakai, T., Saito, Y., Ichimura, Y., Kokubu, T., & Koyama, M. (2004). The effect of back-formulating questions in question answering evaluation. In K. Jarvelin, J. Allen, P. Bruza, & M. Sanderson (Eds.), Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 474-475). (Proceedings of Sheffield SIGIR - Twenty-Seventh Annual International ACM SIGIR Conference on Research and Development in Information Retrieval).