Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment

Woo Suk Choi, Kyoung-Woon On, Yu-Jung Heo, Byoung-Tak Zhang


Abstract
Scene graph is a graph representation that explicitly represents high-level semantic knowledge of an image such as objects, attributes of objects and relationships between objects. Various tasks have been proposed for the scene graph, but the problem is that they have a limited vocabulary and biased information due to their own hypothesis. Therefore, results of each task are not generalizable and difficult to be applied to other down-stream tasks. In this paper, we propose Entity Synset Alignment(ESA), which is a method to create a general scene graph by aligning various semantic knowledge efficiently to solve this bias problem. The ESA uses a large-scale lexical database, WordNet and Intersection of Union (IoU) to align the object labels in multiple scene graphs/semantic knowledge. In experiment, the integrated scene graph is applied to the image-caption retrieval task as a down-stream task. We confirm that integrating multiple scene graphs helps to get better representations of images.
Anthology ID:
2020.alvr-1.2
Volume:
Proceedings of the First Workshop on Advances in Language and Vision Research
Month:
July
Year:
2020
Address:
Online
Editors:
Xin Wang, Jesse Thomason, Ronghang Hu, Xinlei Chen, Peter Anderson, Qi Wu, Asli Celikyilmaz, Jason Baldridge, William Yang Wang
Venue:
ALVR
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7–11
Language:
URL:
https://aclanthology.org/2020.alvr-1.2
DOI:
10.18653/v1/2020.alvr-1.2
Bibkey:
Cite (ACL):
Woo Suk Choi, Kyoung-Woon On, Yu-Jung Heo, and Byoung-Tak Zhang. 2020. Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment. In Proceedings of the First Workshop on Advances in Language and Vision Research, pages 7–11, Online. Association for Computational Linguistics.
Cite (Informal):
Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment (Choi et al., ALVR 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.alvr-1.2.pdf
Video:
 http://slideslive.com/38929758