PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation

Xinyu Hua, Lu Wang


Abstract
Pre-trained Transformers have enabled impressive breakthroughs in generating long and fluent text, yet their outputs are often “rambling” without coherently arranged content. In this work, we present a novel content-controlled text generation framework, PAIR, with planning and iterative refinement, which is built upon a large model, BART. We first adapt the BERT model to automatically construct the content plans, consisting of keyphrase assignments and their corresponding sentence-level positions. The BART model is employed for generation without modifying its structure. We then propose a refinement algorithm to gradually enhance the generation quality within the sequence-to-sequence framework. Evaluation with automatic metrics shows that adding planning consistently improves the generation quality on three distinct domains, with an average of 20 BLEU points and 12 METEOR points improvements. In addition, human judges rate our system outputs to be more relevant and coherent than comparisons without planning.
Anthology ID:
2020.emnlp-main.57
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
781–793
Language:
URL:
https://aclanthology.org/2020.emnlp-main.57
DOI:
10.18653/v1/2020.emnlp-main.57
Bibkey:
Cite (ACL):
Xinyu Hua and Lu Wang. 2020. PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 781–793, Online. Association for Computational Linguistics.
Cite (Informal):
PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation (Hua & Wang, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.57.pdf
Optional supplementary material:
 2020.emnlp-main.57.OptionalSupplementaryMaterial.pdf
Video:
 https://slideslive.com/38939134
Data
New York Times Annotated Corpus