Two phase evaluation for selecting machine translation services

Chunqi Shi, Donghui Lin, Masahiko Shimada, Toru Ishida

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

An increased number of machine translation services are now available. Unfortunately, none of them can provide adequate translation quality for all input sources. This forces the user to select from among the services according to his needs. However, it is tedious and time consuming to perform this manual selection. Our solution, proposed here, is an automatic mechanism that can select the most appropriate machine translation service. Although evaluation methods are available, such as BLEU, NIST, WER, etc., their evaluation results are not unanimous regardless of the translation sources. We proposed a two-phase architecture for selecting translation services. The first phase uses a data-driven classification to allow the most appropriate evaluation method to be selected according to each translation source. The second phase selects the most appropriate machine translation result by the selected evaluation method. We describe the architecture, detail the algorithm, and construct a prototype. Tests show that the proposal yields better translation quality than employing just one machine translation service.

Original languageEnglish
Title of host publicationProceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012
EditorsMehmet Ugur Dogan, Joseph Mariani, Asuncion Moreno, Sara Goggi, Khalid Choukri, Nicoletta Calzolari, Jan Odijk, Thierry Declerck, Bente Maegaard, Stelios Piperidis, Helene Mazo, Olivier Hamon
PublisherEuropean Language Resources Association (ELRA)
Pages1771-1778
Number of pages8
ISBN (Electronic)9782951740877
Publication statusPublished - 2012 Jan 1
Externally publishedYes
Event8th International Conference on Language Resources and Evaluation, LREC 2012 - Istanbul, Turkey
Duration: 2012 May 212012 May 27

Publication series

NameProceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

Other

Other8th International Conference on Language Resources and Evaluation, LREC 2012
CountryTurkey
CityIstanbul
Period12/5/2112/5/27

Fingerprint

evaluation
Evaluation
Machine Translation
Data-driven
Prototype

Keywords

  • Evaluation
  • Machine Translation
  • Service Selection

ASJC Scopus subject areas

  • Linguistics and Language
  • Language and Linguistics
  • Education
  • Library and Information Sciences

Cite this

Shi, C., Lin, D., Shimada, M., & Ishida, T. (2012). Two phase evaluation for selecting machine translation services. In M. U. Dogan, J. Mariani, A. Moreno, S. Goggi, K. Choukri, N. Calzolari, J. Odijk, T. Declerck, B. Maegaard, S. Piperidis, H. Mazo, ... O. Hamon (Eds.), Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012 (pp. 1771-1778). (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012). European Language Resources Association (ELRA).

Two phase evaluation for selecting machine translation services. / Shi, Chunqi; Lin, Donghui; Shimada, Masahiko; Ishida, Toru.

Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. ed. / Mehmet Ugur Dogan; Joseph Mariani; Asuncion Moreno; Sara Goggi; Khalid Choukri; Nicoletta Calzolari; Jan Odijk; Thierry Declerck; Bente Maegaard; Stelios Piperidis; Helene Mazo; Olivier Hamon. European Language Resources Association (ELRA), 2012. p. 1771-1778 (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Shi, C, Lin, D, Shimada, M & Ishida, T 2012, Two phase evaluation for selecting machine translation services. in MU Dogan, J Mariani, A Moreno, S Goggi, K Choukri, N Calzolari, J Odijk, T Declerck, B Maegaard, S Piperidis, H Mazo & O Hamon (eds), Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, European Language Resources Association (ELRA), pp. 1771-1778, 8th International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, 12/5/21.
Shi C, Lin D, Shimada M, Ishida T. Two phase evaluation for selecting machine translation services. In Dogan MU, Mariani J, Moreno A, Goggi S, Choukri K, Calzolari N, Odijk J, Declerck T, Maegaard B, Piperidis S, Mazo H, Hamon O, editors, Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. European Language Resources Association (ELRA). 2012. p. 1771-1778. (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012).
Shi, Chunqi ; Lin, Donghui ; Shimada, Masahiko ; Ishida, Toru. / Two phase evaluation for selecting machine translation services. Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012. editor / Mehmet Ugur Dogan ; Joseph Mariani ; Asuncion Moreno ; Sara Goggi ; Khalid Choukri ; Nicoletta Calzolari ; Jan Odijk ; Thierry Declerck ; Bente Maegaard ; Stelios Piperidis ; Helene Mazo ; Olivier Hamon. European Language Resources Association (ELRA), 2012. pp. 1771-1778 (Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012).
@inproceedings{5367d3942cb0466ab66332feda5f6a1b,
title = "Two phase evaluation for selecting machine translation services",
abstract = "An increased number of machine translation services are now available. Unfortunately, none of them can provide adequate translation quality for all input sources. This forces the user to select from among the services according to his needs. However, it is tedious and time consuming to perform this manual selection. Our solution, proposed here, is an automatic mechanism that can select the most appropriate machine translation service. Although evaluation methods are available, such as BLEU, NIST, WER, etc., their evaluation results are not unanimous regardless of the translation sources. We proposed a two-phase architecture for selecting translation services. The first phase uses a data-driven classification to allow the most appropriate evaluation method to be selected according to each translation source. The second phase selects the most appropriate machine translation result by the selected evaluation method. We describe the architecture, detail the algorithm, and construct a prototype. Tests show that the proposal yields better translation quality than employing just one machine translation service.",
keywords = "Evaluation, Machine Translation, Service Selection",
author = "Chunqi Shi and Donghui Lin and Masahiko Shimada and Toru Ishida",
year = "2012",
month = "1",
day = "1",
language = "English",
series = "Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012",
publisher = "European Language Resources Association (ELRA)",
pages = "1771--1778",
editor = "Dogan, {Mehmet Ugur} and Joseph Mariani and Asuncion Moreno and Sara Goggi and Khalid Choukri and Nicoletta Calzolari and Jan Odijk and Thierry Declerck and Bente Maegaard and Stelios Piperidis and Helene Mazo and Olivier Hamon",
booktitle = "Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012",

}

TY - GEN

T1 - Two phase evaluation for selecting machine translation services

AU - Shi, Chunqi

AU - Lin, Donghui

AU - Shimada, Masahiko

AU - Ishida, Toru

PY - 2012/1/1

Y1 - 2012/1/1

N2 - An increased number of machine translation services are now available. Unfortunately, none of them can provide adequate translation quality for all input sources. This forces the user to select from among the services according to his needs. However, it is tedious and time consuming to perform this manual selection. Our solution, proposed here, is an automatic mechanism that can select the most appropriate machine translation service. Although evaluation methods are available, such as BLEU, NIST, WER, etc., their evaluation results are not unanimous regardless of the translation sources. We proposed a two-phase architecture for selecting translation services. The first phase uses a data-driven classification to allow the most appropriate evaluation method to be selected according to each translation source. The second phase selects the most appropriate machine translation result by the selected evaluation method. We describe the architecture, detail the algorithm, and construct a prototype. Tests show that the proposal yields better translation quality than employing just one machine translation service.

AB - An increased number of machine translation services are now available. Unfortunately, none of them can provide adequate translation quality for all input sources. This forces the user to select from among the services according to his needs. However, it is tedious and time consuming to perform this manual selection. Our solution, proposed here, is an automatic mechanism that can select the most appropriate machine translation service. Although evaluation methods are available, such as BLEU, NIST, WER, etc., their evaluation results are not unanimous regardless of the translation sources. We proposed a two-phase architecture for selecting translation services. The first phase uses a data-driven classification to allow the most appropriate evaluation method to be selected according to each translation source. The second phase selects the most appropriate machine translation result by the selected evaluation method. We describe the architecture, detail the algorithm, and construct a prototype. Tests show that the proposal yields better translation quality than employing just one machine translation service.

KW - Evaluation

KW - Machine Translation

KW - Service Selection

UR - http://www.scopus.com/inward/record.url?scp=85037109889&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85037109889&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85037109889

T3 - Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

SP - 1771

EP - 1778

BT - Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

A2 - Dogan, Mehmet Ugur

A2 - Mariani, Joseph

A2 - Moreno, Asuncion

A2 - Goggi, Sara

A2 - Choukri, Khalid

A2 - Calzolari, Nicoletta

A2 - Odijk, Jan

A2 - Declerck, Thierry

A2 - Maegaard, Bente

A2 - Piperidis, Stelios

A2 - Mazo, Helene

A2 - Hamon, Olivier

PB - European Language Resources Association (ELRA)

ER -