BLEU Evaluation of Machine-Translated English-Croatian Legislation

Sanja Seljan, Marija Brkić, Tomislav Vičić


Abstract
This paper presents work on the evaluation of online available machine translation (MT) service, i.e. Google Translate, for English-Croatian language pair in the domain of legislation. The total set of 200 sentences, for which three reference translations are provided, is divided into short and long sentences. Human evaluation is performed by native speakers, using the criteria of adequacy and fluency. For measuring the reliability of agreement among raters, Fleiss' kappa metric is used. Human evaluation is enriched by error analysis, in order to examine the influence of error types on fluency and adequacy, and to use it in further research. Translation errors are divided into several categories: non-translated words, word omissions, unnecessarily translated words, morphological errors, lexical errors, syntactic errors and incorrect punctuation. The automatic evaluation metric BLEU is calculated with regard to a single and multiple reference translations. System level Pearson's correlation between BLEU scores based on a single and multiple reference translations is given, as well as correlation between short and long sentences BLEU scores, and correlation between the criteria of fluency and adequacy and each error category.
Anthology ID:
L12-1087
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
2143–2148
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/233_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Sanja Seljan, Marija Brkić, and Tomislav Vičić. 2012. BLEU Evaluation of Machine-Translated English-Croatian Legislation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2143–2148, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
BLEU Evaluation of Machine-Translated English-Croatian Legislation (Seljan et al., LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/233_Paper.pdf