Towards Robustifying NLI Models Against Lexical Dataset Biases

Xiang Zhou, Mohit Bansal


Abstract
While deep learning models are making fast progress on the task of Natural Language Inference, recent studies have also shown that these models achieve high accuracy by exploiting several dataset biases, and without deep understanding of the language semantics. Using contradiction-word bias and word-overlapping bias as our two bias examples, this paper explores both data-level and model-level debiasing methods to robustify models against lexical dataset biases. First, we debias the dataset through data augmentation and enhancement, but show that the model bias cannot be fully removed via this method. Next, we also compare two ways of directly debiasing the model without knowing what the dataset biases are in advance. The first approach aims to remove the label bias at the embedding level. The second approach employs a bag-of-words sub-model to capture the features that are likely to exploit the bias and prevents the original model from learning these biased features by forcing orthogonality between these two sub-models. We performed evaluations on new balanced datasets extracted from the original MNLI dataset as well as the NLI stress tests, and show that the orthogonality approach is better at debiasing the model while maintaining competitive overall accuracy.
Anthology ID:
2020.acl-main.773
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8759–8771
Language:
URL:
https://aclanthology.org/2020.acl-main.773
DOI:
10.18653/v1/2020.acl-main.773
Bibkey:
Cite (ACL):
Xiang Zhou and Mohit Bansal. 2020. Towards Robustifying NLI Models Against Lexical Dataset Biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759–8771, Online. Association for Computational Linguistics.
Cite (Informal):
Towards Robustifying NLI Models Against Lexical Dataset Biases (Zhou & Bansal, ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.773.pdf
Video:
 http://slideslive.com/38929270
Code
 owenzx/LexicalDebias-ACL2020
Data
MultiNLI