The MUCOW word sense disambiguation test suite at WMT 2020

Yves Scherrer, Alessandro Raganato, Jörg Tiedemann


Abstract
This paper reports on our participation with the MUCOW test suite at the WMT 2020 news translation task. We introduced MUCOW at WMT 2019 to measure the ability of MT systems to perform word sense disambiguation (WSD), i.e., to translate an ambiguous word with its correct sense. MUCOW is created automatically using existing resources, and the evaluation process is also entirely automated. We evaluate all participating systems of the language pairs English -> Czech, English -> German, and English -> Russian and compare the results with those obtained at WMT 2019. While current NMT systems are fairly good at handling ambiguous source words, we could not identify any substantial progress - at least to the extent that it is measurable by the MUCOW method - in that area over the last year.
Anthology ID:
2020.wmt-1.40
Volume:
Proceedings of the Fifth Conference on Machine Translation
Month:
November
Year:
2020
Address:
Online
Editors:
Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri
Venue:
WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
365–370
Language:
URL:
https://aclanthology.org/2020.wmt-1.40
DOI:
Bibkey:
Cite (ACL):
Yves Scherrer, Alessandro Raganato, and Jörg Tiedemann. 2020. The MUCOW word sense disambiguation test suite at WMT 2020. In Proceedings of the Fifth Conference on Machine Translation, pages 365–370, Online. Association for Computational Linguistics.
Cite (Informal):
The MUCOW word sense disambiguation test suite at WMT 2020 (Scherrer et al., WMT 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.wmt-1.40.pdf
Video:
 https://slideslive.com/38939567
Code
 Helsinki-NLP/MuCoW