Depth Growing for Neural Machine Translation

Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, Tie-Yan Liu


Abstract
While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of the neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even drop in performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT14 EnglishGerman and EnglishFrench translation tasks.
Anthology ID:
P19-1558
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Anna Korhonen, David Traum, Lluís Màrquez
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5558–5563
Language:
URL:
https://aclanthology.org/P19-1558
DOI:
10.18653/v1/P19-1558
Bibkey:
Cite (ACL):
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Depth Growing for Neural Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5558–5563, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Depth Growing for Neural Machine Translation (Wu et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1558.pdf
Supplementary:
 P19-1558.Supplementary.zip
Code
 apeterswu/Depth_Growing_NMT
Data
WMT 2014