Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA

Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, Michael Elhadad


Abstract
Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. Contrast sets (Gardneret al., 2020) quantify this phenomenon by perturbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requiring intensive annotation effort, we present a novel method which leverages rich semantic input representation to automatically generate contrast sets for the visual question answering task. Our method computes the answer of perturbed questions, thus vastly reducing annotation cost and enabling thorough evaluation of models’ performance on various semantic aspects (e.g., spatial or relational reasoning). We demonstrate the effectiveness of our approach on the GQA dataset and its semantic scene graph image representation. We find that, despite GQA’s compositionality and carefully balanced label distribution, two high-performing models drop 13-17% in accuracy compared to the original test set. Finally, we show that our automatic perturbation can be applied to the training set to mitigate the degradation in performance, opening the door to more robust models.
Anthology ID:
2021.naacl-main.9
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
94–105
Language:
URL:
https://aclanthology.org/2021.naacl-main.9
DOI:
10.18653/v1/2021.naacl-main.9
Bibkey:
Cite (ACL):
Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad. 2021. Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 94–105, Online. Association for Computational Linguistics.
Cite (Informal):
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA (Bitton et al., NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.9.pdf
Video:
 https://aclanthology.org/2021.naacl-main.9.mp4
Code
 yonatanbitton/AutoGenOfContrastSetsFromSceneGraphs +  additional community code
Data
GQA