Structural and Functional Decomposition for Personality Image Captioning in a Communication Game

Minh Thu Nguyen, Duy Phung, Minh Hoai, Thien Huu Nguyen


Abstract
Personality image captioning (PIC) aims to describe an image with a natural language caption given a personality trait. In this work, we introduce a novel formulation for PIC based on a communication game between a speaker and a listener. The speaker attempts to generate natural language captions while the listener encourages the generated captions to contain discriminative information about the input images and personality traits. In this way, we expect that the generated captions can be improved to naturally represent the images and express the traits. In addition, we propose to adapt the language model GPT2 to perform caption generation for PIC. This enables the speaker and listener to benefit from the language encoding capacity of GPT2. Our experiments show that the proposed model achieves the state-of-the-art performance for PIC.
Anthology ID:
2020.findings-emnlp.411
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4587–4593
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.411
DOI:
10.18653/v1/2020.findings-emnlp.411
Bibkey:
Cite (ACL):
Minh Thu Nguyen, Duy Phung, Minh Hoai, and Thien Huu Nguyen. 2020. Structural and Functional Decomposition for Personality Image Captioning in a Communication Game. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4587–4593, Online. Association for Computational Linguistics.
Cite (Informal):
Structural and Functional Decomposition for Personality Image Captioning in a Communication Game (Nguyen et al., Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.411.pdf
Optional supplementary material:
 2020.findings-emnlp.411.OptionalSupplementaryMaterial.zip