Representation of Linguistic Form and Function in Recurrent Neural Networks

Ákos Kádár, Grzegorz Chrupała, Afra Alishahi


Abstract
We present novel methods for analyzing the activation patterns of recurrent neural networks from a linguistic point of view and explore the types of linguistic structure they learn. As a case study, we use a standard standalone language model, and a multi-task gated recurrent network architecture consisting of two parallel pathways with shared word embeddings: The Visual pathway is trained on predicting the representations of the visual scene corresponding to an input sentence, and the Textual pathway is trained to predict the next word in the same sentence. We propose a method for estimating the amount of contribution of individual tokens in the input to the final prediction of the networks. Using this method, we show that the Visual pathway pays selective attention to lexical categories and grammatical functions that carry semantic information, and learns to treat word types differently depending on their grammatical function and their position in the sequential structure of the sentence. In contrast, the language models are comparatively more sensitive to words with a syntactic function. Further analysis of the most informative n-gram contexts for each model shows that in comparison with the Visual pathway, the language models react more strongly to abstract contexts that represent syntactic constructions.
Anthology ID:
J17-4003
Volume:
Computational Linguistics, Volume 43, Issue 4 - December 2017
Month:
December
Year:
2017
Address:
Cambridge, MA
Venue:
CL
SIG:
Publisher:
MIT Press
Note:
Pages:
761–780
Language:
URL:
https://aclanthology.org/J17-4003
DOI:
10.1162/COLI_a_00300
Bibkey:
Cite (ACL):
Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2017. Representation of Linguistic Form and Function in Recurrent Neural Networks. Computational Linguistics, 43(4):761–780.
Cite (Informal):
Representation of Linguistic Form and Function in Recurrent Neural Networks (Kádár et al., CL 2017)
Copy Citation:
PDF:
https://aclanthology.org/J17-4003.pdf
Data
MS COCO