Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations

Jeff Da, Jungo Kasai


Abstract
Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models. Thus, we investigate and challenge several aspects of BERT’s commonsense representation abilities. First, we probe BERT’s ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas. Next, we show that, by augmenting BERT’s pretraining data with additional data related to the deficient attributes, we are able to improve performance on a downstream commonsense reasoning task while using a minimal amount of data. Finally, we develop a method of fine-tuning knowledge graphs embeddings alongside BERT and show the continued importance of explicit knowledge graphs.
Anthology ID:
D19-6001
Volume:
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Simon Ostermann, Sheng Zhang, Michael Roth, Peter Clark
Venue:
WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–12
Language:
URL:
https://aclanthology.org/D19-6001
DOI:
10.18653/v1/D19-6001
Bibkey:
Cite (ACL):
Jeff Da and Jungo Kasai. 2019. Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 1–12, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (Da & Kasai, 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-6001.pdf
Data
ATOMICMCScriptRACEWebChild