Reference-Aware Language Models

Zichao Yang, Phil Blunsom, Chris Dyer, Wang Ling


Abstract
We propose a general class of language models that treat reference as discrete stochastic latent variables. This decision allows for the creation of entity mentions by accessing external databases of referents (required by, e.g., dialogue generation) or past internal state (required to explicitly model coreferentiality). Beyond simple copying, our coreference model can additionally refer to a referent using varied mention forms (e.g., a reference to “Jane” can be realized as “she”), a characteristic feature of reference in natural languages. Experiments on three representative applications show our model variants outperform models based on deterministic attention and standard language modeling baselines.
Anthology ID:
D17-1197
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1850–1859
Language:
URL:
https://aclanthology.org/D17-1197
DOI:
10.18653/v1/D17-1197
Bibkey:
Cite (ACL):
Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2017. Reference-Aware Language Models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1850–1859, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Reference-Aware Language Models (Yang et al., EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1197.pdf
Attachment:
 D17-1197.Attachment.pdf