The Effect of Translationese in Machine Translation Test Sets

Mike Zhang, Antonio Toral


Abstract
The effect of translationese has been studied in the field of machine translation (MT), mostly with respect to training data. We study in depth the effect of translationese on test data, using the test sets from the last three editions of WMT’s news shared task, containing 17 translation directions. We show evidence that (i) the use of translationese in test sets results in inflated human evaluation scores for MT systems; (ii) in some cases system rankings do change and (iii) the impact translationese has on a translation direction is inversely correlated to the translation quality attainable by state-of-the-art MT systems for that direction.
Anthology ID:
W19-5208
Volume:
Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)
Month:
August
Year:
2019
Address:
Florence, Italy
Editors:
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, Karin Verspoor
Venue:
WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
73–81
Language:
URL:
https://aclanthology.org/W19-5208
DOI:
10.18653/v1/W19-5208
Bibkey:
Cite (ACL):
Mike Zhang and Antonio Toral. 2019. The Effect of Translationese in Machine Translation Test Sets. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 73–81, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
The Effect of Translationese in Machine Translation Test Sets (Zhang & Toral, WMT 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-5208.pdf
Supplementary:
 W19-5208.Supplementary.zip
Presentation:
 W19-5208.Presentation.pdf
Code
 jjzha/translationese
Data
WMT 2016