Scene Graph Parsing as Dependency Parsing

Yu-Siang Wang, Chenxi Liu, Xiaohui Zeng, Alan Yuille


Abstract
In this paper, we study the problem of parsing structured knowledge graphs from textual descriptions. In particular, we consider the scene graph representation that considers objects together with their attributes and relations: this representation has been proved useful across a variety of vision and language applications. We begin by introducing an alternative but equivalent edge-centric view of scene graphs that connect to dependency parses. Together with a careful redesign of label and action space, we combine the two-stage pipeline used in prior work (generic dependency parsing followed by simple post-processing) into one, enabling end-to-end training. The scene graphs generated by our learned neural dependency parser achieve an F-score similarity of 49.67% to ground truth graphs on our evaluation set, surpassing best previous approaches by 5%. We further demonstrate the effectiveness of our learned parser on image retrieval applications.
Anthology ID:
N18-1037
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
397–407
Language:
URL:
https://aclanthology.org/N18-1037
DOI:
10.18653/v1/N18-1037
Bibkey:
Cite (ACL):
Yu-Siang Wang, Chenxi Liu, Xiaohui Zeng, and Alan Yuille. 2018. Scene Graph Parsing as Dependency Parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 397–407, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Scene Graph Parsing as Dependency Parsing (Wang et al., NAACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/N18-1037.pdf
Video:
 https://aclanthology.org/N18-1037.mp4
Code
 Yusics/bist-parser +  additional community code
Data
MS COCOVisual GenomeVisual Question Answering