Using multiple edit distances to automatically grade outputs from machine translation systems

Yasuhiro Akiba, Kenji Imamura, Eiichiro Sumita, Hiromi Nakaiwa, Seiichi Yamamoto, Hiroshi G. Okuno

Research output: Contribution to journalArticle

Abstract

This paper addresses the challenging problem of automatically evaluating output from machine translation (MT) systems that are subsystems of speech-to-speech MT (SSMT) systems. Conventional automatic MT evaluation methods include BLEU, which MT researchers have frequently used. However, BLEU has two drawbacks in SSMT evaluation. First, BLEU assesses errors lightly at the beginning of translations and heavily in the middle, even though its assessments should be independent of position. Second, BLEU lacks tolerance in accepting colloquial sentences with small errors, although such errors do not prevent us from continuing an SSMT-mediated conversation. In this paper, the authors report a new evaluation method called "gRader based on Edit Distances (RED)" that automatically grades each MT output by using a decision tree (DT). The DT is learned from training data that are encoded by using multiple edit distances, that is, normal edit distance (ED) defined by insertion, deletion, and replacement, as well as its extensions. The use of multiple edit distances allows more tolerance than either ED or BLEU. Each evaluated MT output is assigned a grade by using the DT. RED and BLEU were compared for the task of evaluating MT systems of varying quality on ATR's Basic Travel Expression Corpus (BTEC). Experimental results show that RED significantly outperforms BLEU.

Original languageEnglish
Pages (from-to)393-401
Number of pages9
JournalIEEE Transactions on Audio, Speech and Language Processing
Volume14
Issue number2
DOIs
Publication statusPublished - 2006
Externally publishedYes

Keywords

  • BLEU
  • Decision tree (DT)
  • Edit distances (EDs)
  • Machine translation evaluation
  • mWER
  • Reference translations

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Acoustics and Ultrasonics

Fingerprint Dive into the research topics of 'Using multiple edit distances to automatically grade outputs from machine translation systems'. Together they form a unique fingerprint.

  • Cite this