Multi-modal Multi-label Emotion Detection with Modality and Label Dependence

Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, Guodong Zhou


Abstract
As an important research issue in the natural language processing community, multi-label emotion detection has been drawing more and more attention in the last few years. However, almost all existing studies focus on one modality (e.g., textual modality). In this paper, we focus on multi-label emotion detection in a multi-modal scenario. In this scenario, we need to consider both the dependence among different labels (label dependence) and the dependence between each predicting label and different modalities (modality dependence). Particularly, we propose a multi-modal sequence-to-set approach to effectively model both kinds of dependence in multi-modal multi-label emotion detection. The detailed evaluation demonstrates the effectiveness of our approach.
Anthology ID:
2020.emnlp-main.291
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3584–3593
Language:
URL:
https://aclanthology.org/2020.emnlp-main.291
DOI:
10.18653/v1/2020.emnlp-main.291
Bibkey:
Cite (ACL):
Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. 2020. Multi-modal Multi-label Emotion Detection with Modality and Label Dependence. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3584–3593, Online. Association for Computational Linguistics.
Cite (Informal):
Multi-modal Multi-label Emotion Detection with Modality and Label Dependence (Zhang et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.291.pdf
Video:
 https://slideslive.com/38938701