Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition

Angli Liu, Jingfei Du, Veselin Stoyanov


Abstract
Traditional language models are unable to efficiently model entity names observed in text. All but the most popular named entities appear infrequently in text providing insufficient context. Recent efforts have recognized that context can be generalized between entity names that share the same type (e.g., person or location) and have equipped language models with an access to external knowledge base (KB). Our Knowledge-Augmented Language Model (KALM) continues this line of work by augmenting a traditional model with a KB. Unlike previous methods, however, we train with an end-to-end predictive objective optimizing the perplexity of text. We do not require any additional information such as named entity tags. In addition to improving language modeling performance, KALM learns to recognize named entities in an entirely unsupervised way by using entity type information latent in the model. On a Named Entity Recognition (NER) task, KALM achieves performance comparable with state-of-the-art supervised models. Our work demonstrates that named entities (and possibly other types of world knowledge) can be modeled successfully using predictive learning and training on large corpora of text without any additional information.
Anthology ID:
N19-1117
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1142–1150
Language:
URL:
https://aclanthology.org/N19-1117
DOI:
10.18653/v1/N19-1117
Bibkey:
Cite (ACL):
Angli Liu, Jingfei Du, and Veselin Stoyanov. 2019. Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1142–1150, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition (Liu et al., NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1117.pdf
Data
CoNLL 2003WikiText-2