Abstract
The recent popularity of machine translation has increased the demand for the evaluation of translations. However, the traditional evaluation approach, manual checking by a bilingual professional, is too expensive and too slow. In this study, we confirm the feasibility of crowdsourcing by analyzing the accuracy of crowdsourcing translation evaluations. We compare crowdsourcing scores to professional scores with regard to three metrics: translation-score, sentence-score, and system-score. A Chinese to English translation evaluation task was designed using around the NTCIR-9 PATENT parallel corpus with the goal being 5-range evaluations of adequacy and fluency. The experiment shows that the average score of crowdsource workers well matches professional evaluation results. The system-score comparison strongly indicates that crowdsourcing can be used to find the best translation system given the input of 10 source sentence.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014 |
| Editors | Nicoletta Calzolari, Khalid Choukri, Sara Goggi, Thierry Declerck, Joseph Mariani, Bente Maegaard, Asuncion Moreno, Jan Odijk, Helene Mazo, Stelios Piperidis, Hrafn Loftsson |
| Publisher | European Language Resources Association (ELRA) |
| Pages | 3456-3463 |
| Number of pages | 8 |
| ISBN (Electronic) | 9782951740884 |
| Publication status | Published - 2014 Jan 1 |
| Externally published | Yes |
| Event | 9th International Conference on Language Resources and Evaluation, LREC 2014 - Reykjavik, Iceland Duration: 2014 May 26 → 2014 May 31 |
Publication series
| Name | Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014 |
|---|
Other
| Other | 9th International Conference on Language Resources and Evaluation, LREC 2014 |
|---|---|
| Country | Iceland |
| City | Reykjavik |
| Period | 14/5/26 → 14/5/31 |
Fingerprint
Keywords
- Crowdsourcing
- Evaluation
- Machine translation
ASJC Scopus subject areas
- Linguistics and Language
- Library and Information Sciences
- Education
- Language and Linguistics
Cite this
Crowdsourcing for evaluating machine translation quality. / Goto, Shinsuke; Lin, Donghui; Ishida, Toru.
Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014. ed. / Nicoletta Calzolari; Khalid Choukri; Sara Goggi; Thierry Declerck; Joseph Mariani; Bente Maegaard; Asuncion Moreno; Jan Odijk; Helene Mazo; Stelios Piperidis; Hrafn Loftsson. European Language Resources Association (ELRA), 2014. p. 3456-3463 (Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Crowdsourcing for evaluating machine translation quality
AU - Goto, Shinsuke
AU - Lin, Donghui
AU - Ishida, Toru
PY - 2014/1/1
Y1 - 2014/1/1
N2 - The recent popularity of machine translation has increased the demand for the evaluation of translations. However, the traditional evaluation approach, manual checking by a bilingual professional, is too expensive and too slow. In this study, we confirm the feasibility of crowdsourcing by analyzing the accuracy of crowdsourcing translation evaluations. We compare crowdsourcing scores to professional scores with regard to three metrics: translation-score, sentence-score, and system-score. A Chinese to English translation evaluation task was designed using around the NTCIR-9 PATENT parallel corpus with the goal being 5-range evaluations of adequacy and fluency. The experiment shows that the average score of crowdsource workers well matches professional evaluation results. The system-score comparison strongly indicates that crowdsourcing can be used to find the best translation system given the input of 10 source sentence.
AB - The recent popularity of machine translation has increased the demand for the evaluation of translations. However, the traditional evaluation approach, manual checking by a bilingual professional, is too expensive and too slow. In this study, we confirm the feasibility of crowdsourcing by analyzing the accuracy of crowdsourcing translation evaluations. We compare crowdsourcing scores to professional scores with regard to three metrics: translation-score, sentence-score, and system-score. A Chinese to English translation evaluation task was designed using around the NTCIR-9 PATENT parallel corpus with the goal being 5-range evaluations of adequacy and fluency. The experiment shows that the average score of crowdsource workers well matches professional evaluation results. The system-score comparison strongly indicates that crowdsourcing can be used to find the best translation system given the input of 10 source sentence.
KW - Crowdsourcing
KW - Evaluation
KW - Machine translation
UR - http://www.scopus.com/inward/record.url?scp=85026825744&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85026825744&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85026825744
T3 - Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014
SP - 3456
EP - 3463
BT - Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014
A2 - Calzolari, Nicoletta
A2 - Choukri, Khalid
A2 - Goggi, Sara
A2 - Declerck, Thierry
A2 - Mariani, Joseph
A2 - Maegaard, Bente
A2 - Moreno, Asuncion
A2 - Odijk, Jan
A2 - Mazo, Helene
A2 - Piperidis, Stelios
A2 - Loftsson, Hrafn
PB - European Language Resources Association (ELRA)
ER -