A Formal Analysis of Multimodal Referring Strategies Under Common Ground

Nikhil Krishnaswamy, James Pustejovsky


Abstract
In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions. In doing so, we expose some striking formal semantic properties of the interactions between gesture and language, conditioned on the introduction of content into the common ground between the (computational) speaker and (human) viewer, and demonstrate how these formal features can contribute to training better models to predict viewer judgment of referring expressions, and potentially to the generation of more natural and informative referring expressions.
Anthology ID:
2020.lrec-1.725
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
5919–5927
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.725
DOI:
Bibkey:
Cite (ACL):
Nikhil Krishnaswamy and James Pustejovsky. 2020. A Formal Analysis of Multimodal Referring Strategies Under Common Ground. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5919–5927, Marseille, France. European Language Resources Association.
Cite (Informal):
A Formal Analysis of Multimodal Referring Strategies Under Common Ground (Krishnaswamy & Pustejovsky, LREC 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.lrec-1.725.pdf