Bilingual lexicon induction

Effortless evaluation of word alignment tools and production of resources for improbable language pairs

Adrien Lardilleux, Julien Gosme, Yves Lepage

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

In this paper, we present a simple protocol to evaluate word aligners on bilingual lexicon induction tasks from parallel corpora. Rather than resorting to gold standards, it relies on a comparison of the outputs of word aligners against a reference bilingual lexicon. The quality of this reference bilingual lexicon does not need to be particularly high, because evaluation quality is ensured by systematically filtering this reference lexicon with the parallel corpus the word aligners are trained on. We perform a comparison of three freely available word aligners on numerous language pairs from the Bible parallel corpus (Resnik et al., 1999): MGIZA++ (Gao and Vogel, 2008), BerkeleyAligner (Liang et al., 2006), and Anymalign (Lardilleux and Lepage, 2009). We then select the most appropriate one to produce bilingual lexicons for all language pairs of this corpus. These involve Cebuano, Chinese, Danish, English, Finnish, French, Greek, Indonesian, Latin, Spanish, Swedish, and Vietnamese. The 66 resulting lexicons are made freely available.

Original languageEnglish
Title of host publicationProceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010
PublisherEuropean Language Resources Association (ELRA)
Pages252-256
Number of pages5
ISBN (Electronic)2951740867, 9782951740860
Publication statusPublished - 2010 Jan 1
Externally publishedYes
Event7th International Conference on Language Resources and Evaluation, LREC 2010 - Valletta, Malta
Duration: 2010 May 172010 May 23

Other

Other7th International Conference on Language Resources and Evaluation, LREC 2010
CountryMalta
CityValletta
Period10/5/1710/5/23

Fingerprint

induction
gold standard
bible
language
evaluation
resources
Bilingual Lexicon
Resources
Language
Alignment
Evaluation
Induction
Parallel Corpora
Lexicon

ASJC Scopus subject areas

  • Education
  • Library and Information Sciences
  • Linguistics and Language
  • Language and Linguistics

Cite this

Lardilleux, A., Gosme, J., & Lepage, Y. (2010). Bilingual lexicon induction: Effortless evaluation of word alignment tools and production of resources for improbable language pairs. In Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010 (pp. 252-256). European Language Resources Association (ELRA).

Bilingual lexicon induction : Effortless evaluation of word alignment tools and production of resources for improbable language pairs. / Lardilleux, Adrien; Gosme, Julien; Lepage, Yves.

Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010. European Language Resources Association (ELRA), 2010. p. 252-256.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lardilleux, A, Gosme, J & Lepage, Y 2010, Bilingual lexicon induction: Effortless evaluation of word alignment tools and production of resources for improbable language pairs. in Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010. European Language Resources Association (ELRA), pp. 252-256, 7th International Conference on Language Resources and Evaluation, LREC 2010, Valletta, Malta, 10/5/17.
Lardilleux A, Gosme J, Lepage Y. Bilingual lexicon induction: Effortless evaluation of word alignment tools and production of resources for improbable language pairs. In Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010. European Language Resources Association (ELRA). 2010. p. 252-256
Lardilleux, Adrien ; Gosme, Julien ; Lepage, Yves. / Bilingual lexicon induction : Effortless evaluation of word alignment tools and production of resources for improbable language pairs. Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010. European Language Resources Association (ELRA), 2010. pp. 252-256
@inproceedings{94c909bf86614a27b8c062698c8f1ea4,
title = "Bilingual lexicon induction: Effortless evaluation of word alignment tools and production of resources for improbable language pairs",
abstract = "In this paper, we present a simple protocol to evaluate word aligners on bilingual lexicon induction tasks from parallel corpora. Rather than resorting to gold standards, it relies on a comparison of the outputs of word aligners against a reference bilingual lexicon. The quality of this reference bilingual lexicon does not need to be particularly high, because evaluation quality is ensured by systematically filtering this reference lexicon with the parallel corpus the word aligners are trained on. We perform a comparison of three freely available word aligners on numerous language pairs from the Bible parallel corpus (Resnik et al., 1999): MGIZA++ (Gao and Vogel, 2008), BerkeleyAligner (Liang et al., 2006), and Anymalign (Lardilleux and Lepage, 2009). We then select the most appropriate one to produce bilingual lexicons for all language pairs of this corpus. These involve Cebuano, Chinese, Danish, English, Finnish, French, Greek, Indonesian, Latin, Spanish, Swedish, and Vietnamese. The 66 resulting lexicons are made freely available.",
author = "Adrien Lardilleux and Julien Gosme and Yves Lepage",
year = "2010",
month = "1",
day = "1",
language = "English",
pages = "252--256",
booktitle = "Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010",
publisher = "European Language Resources Association (ELRA)",

}

TY - GEN

T1 - Bilingual lexicon induction

T2 - Effortless evaluation of word alignment tools and production of resources for improbable language pairs

AU - Lardilleux, Adrien

AU - Gosme, Julien

AU - Lepage, Yves

PY - 2010/1/1

Y1 - 2010/1/1

N2 - In this paper, we present a simple protocol to evaluate word aligners on bilingual lexicon induction tasks from parallel corpora. Rather than resorting to gold standards, it relies on a comparison of the outputs of word aligners against a reference bilingual lexicon. The quality of this reference bilingual lexicon does not need to be particularly high, because evaluation quality is ensured by systematically filtering this reference lexicon with the parallel corpus the word aligners are trained on. We perform a comparison of three freely available word aligners on numerous language pairs from the Bible parallel corpus (Resnik et al., 1999): MGIZA++ (Gao and Vogel, 2008), BerkeleyAligner (Liang et al., 2006), and Anymalign (Lardilleux and Lepage, 2009). We then select the most appropriate one to produce bilingual lexicons for all language pairs of this corpus. These involve Cebuano, Chinese, Danish, English, Finnish, French, Greek, Indonesian, Latin, Spanish, Swedish, and Vietnamese. The 66 resulting lexicons are made freely available.

AB - In this paper, we present a simple protocol to evaluate word aligners on bilingual lexicon induction tasks from parallel corpora. Rather than resorting to gold standards, it relies on a comparison of the outputs of word aligners against a reference bilingual lexicon. The quality of this reference bilingual lexicon does not need to be particularly high, because evaluation quality is ensured by systematically filtering this reference lexicon with the parallel corpus the word aligners are trained on. We perform a comparison of three freely available word aligners on numerous language pairs from the Bible parallel corpus (Resnik et al., 1999): MGIZA++ (Gao and Vogel, 2008), BerkeleyAligner (Liang et al., 2006), and Anymalign (Lardilleux and Lepage, 2009). We then select the most appropriate one to produce bilingual lexicons for all language pairs of this corpus. These involve Cebuano, Chinese, Danish, English, Finnish, French, Greek, Indonesian, Latin, Spanish, Swedish, and Vietnamese. The 66 resulting lexicons are made freely available.

UR - http://www.scopus.com/inward/record.url?scp=85037066948&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85037066948&partnerID=8YFLogxK

M3 - Conference contribution

SP - 252

EP - 256

BT - Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010

PB - European Language Resources Association (ELRA)

ER -