Predicting Reference: What do Language Models Learn about Discourse Models?

Shiva Upadhye, Leon Bergen, Andrew Kehler


Abstract
Whereas there is a growing literature that probes neural language models to assess the degree to which they have latently acquired grammatical knowledge, little if any research has investigated their acquisition of discourse modeling ability. We address this question by drawing on a rich psycholinguistic literature that has established how different contexts affect referential biases concerning who is likely to be referred to next. The results reveal that, for the most part, the prediction behavior of neural language models does not resemble that of human language users.
Anthology ID:
2020.emnlp-main.70
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
977–982
Language:
URL:
https://aclanthology.org/2020.emnlp-main.70
DOI:
10.18653/v1/2020.emnlp-main.70
Bibkey:
Cite (ACL):
Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020. Predicting Reference: What do Language Models Learn about Discourse Models?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 977–982, Online. Association for Computational Linguistics.
Cite (Informal):
Predicting Reference: What do Language Models Learn about Discourse Models? (Upadhye et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.70.pdf
Video:
 https://slideslive.com/38939162