Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer

Lee Hsun Hsieh, Yang-Yin Lee, Ee-Peng Lim


Abstract
Pretrained using large amount of data, autoregressive language models are able to generate high quality sequences. However, these models do not perform well under hard lexical constraints as they lack fine control of content generation process. Progressive insertion based transformers can overcome the above limitation and efficiently generate a sequence in parallel given some input tokens as constraint. These transformers however may fail to support hard lexical constraints as their generation process is more likely to terminate prematurely. The paper analyses such early termination problems and proposes the ENtity CONstrained insertion TransformER(ENCONTER), a new insertion transformer that addresses the above pitfall without compromising much generation efficiency. We introduce a new training strategy that considers predefined hard lexical constraints (e.g., entities to be included in the generated sequence). Our experiments show that ENCONTER outperforms other baseline models in several performance metrics rendering it more suitable in practical applications.
Anthology ID:
2021.eacl-main.313
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3590–3599
Language:
URL:
https://aclanthology.org/2021.eacl-main.313
DOI:
10.18653/v1/2021.eacl-main.313
Bibkey:
Cite (ACL):
Lee Hsun Hsieh, Yang-Yin Lee, and Ee-Peng Lim. 2021. Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3590–3599, Online. Association for Computational Linguistics.
Cite (Informal):
Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer (Hsieh et al., EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.313.pdf
Code
 LARC-CMU-SMU/Enconter
Data
CoNLL 2003