Pre-trained language model representations for language generation

Sergey Edunov, Alexei Baevski, Michael Auli


Abstract
Pre-trained language model representations have been successful in a wide range of language understanding tasks. In this paper, we examine different strategies to integrate pre-trained representations into sequence to sequence models and apply it to neural machine translation and abstractive summarization. We find that pre-trained representations are most effective when added to the encoder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe improvements when millions of sentence-pairs are available. Finally, on abstractive summarization we achieve a new state of the art on the full text version of CNN/DailyMail.
Anthology ID:
N19-1409
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4052–4059
Language:
URL:
https://aclanthology.org/N19-1409
DOI:
10.18653/v1/N19-1409
Bibkey:
Cite (ACL):
Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4052–4059, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Pre-trained language model representations for language generation (Edunov et al., NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1409.pdf
Supplementary:
 N19-1409.Supplementary.pdf
Video:
 https://aclanthology.org/N19-1409.mp4
Code
 pytorch/fairseq
Data
CNN/Daily Mail