Topically Driven Neural Language Model

Jey Han Lau, Timothy Baldwin, Trevor Cohn


Abstract
Language models are typically applied at the sentence level, without access to the broader document context. We present a neural language model that incorporates document context in the form of a topic model-like architecture, thus providing a succinct representation of the broader document context outside of the current sentence. Experiments over a range of datasets demonstrate that our model outperforms a pure sentence-based model in terms of language model perplexity, and leads to topics that are potentially more coherent than those produced by a standard LDA topic model. Our model also has the ability to generate related sentences for a topic, providing another way to interpret topics.
Anthology ID:
P17-1033
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Editors:
Regina Barzilay, Min-Yen Kan
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
355–365
Language:
URL:
https://aclanthology.org/P17-1033
DOI:
10.18653/v1/P17-1033
Bibkey:
Cite (ACL):
Jey Han Lau, Timothy Baldwin, and Trevor Cohn. 2017. Topically Driven Neural Language Model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 355–365, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Topically Driven Neural Language Model (Lau et al., ACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/P17-1033.pdf
Note:
 P17-1033.Notes.zip
Video:
 https://aclanthology.org/P17-1033.mp4
Code
 jhlau/topically-driven-language-model
Data
IMDb Movie Reviews