IRT-based aggregation model of crowdsourced pairwise comparisons for evaluating machine translations

Naoki Otani, Toshiaki Nakazawa, Daisuke Kawahara, Sadao Kurohashi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Recent work on machine translation has used crowdsourcing to reduce costs of manual evaluations. However, crowdsourced judgments are often biased and inaccurate. In this paper, we present a statistical model that aggregates many manual pairwise comparisons to robustly measure a machine translation system's performance. Our method applies graded response model from item response theory (IRT), which was originally developed for academic tests. We conducted experiments on a public dataset from the Workshop on Statistical Machine Translation 2013, and found that our approach resulted in highly interpretable estimates and was less affected by noisy judges than previously proposed methods.

Original languageEnglish
Title of host publicationEMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings
PublisherAssociation for Computational Linguistics (ACL)
Pages511-520
Number of pages10
ISBN (Electronic)9781945626258
Publication statusPublished - 2016
Externally publishedYes
Event2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016 - Austin, United States
Duration: 2016 Nov 12016 Nov 5

Publication series

NameEMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings

Conference

Conference2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016
CountryUnited States
CityAustin
Period16/11/116/11/5

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'IRT-based aggregation model of crowdsourced pairwise comparisons for evaluating machine translations'. Together they form a unique fingerprint.

  • Cite this

    Otani, N., Nakazawa, T., Kawahara, D., & Kurohashi, S. (2016). IRT-based aggregation model of crowdsourced pairwise comparisons for evaluating machine translations. In EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 511-520). (EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings). Association for Computational Linguistics (ACL).