Multimodal Logical Inference System for Visual-Textual Entailment

Riko Suzuki, Hitomi Yanaka, Masashi Yoshikawa, Koji Mineshima, Daisuke Bekki


Abstract
A large amount of research about multimodal inference across text and vision has been recently developed to obtain visually grounded word and sentence representations. In this paper, we use logic-based representations as unified meaning representations for texts and images and present an unsupervised multimodal logical inference system that can effectively prove entailment relations between them. We show that by combining semantic parsing and theorem proving, the system can handle semantically complex sentences for visual-textual inference.
Anthology ID:
P19-2054
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Fernando Alva-Manchego, Eunsol Choi, Daniel Khashabi
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
386–392
Language:
URL:
https://aclanthology.org/P19-2054
DOI:
10.18653/v1/P19-2054
Bibkey:
Cite (ACL):
Riko Suzuki, Hitomi Yanaka, Masashi Yoshikawa, Koji Mineshima, and Daisuke Bekki. 2019. Multimodal Logical Inference System for Visual-Textual Entailment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 386–392, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Multimodal Logical Inference System for Visual-Textual Entailment (Suzuki et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-2054.pdf
Data
Visual GenomeVisual Question Answering