The Lexical Gap: An Improved Measure of Automated Image Description Quality

Austin Kershaw, Miroslaw Bober


Abstract
The challenge of automatically describing images and videos has stimulated much research in Computer Vision and Natural Language Processing. In order to test the semantic abilities of new algorithms, we need reliable and objective ways of measuring progress. We show that standard evaluation measures do not take into account the semantic richness of a description, and give the impression that sparse machine descriptions outperform rich human descriptions. We introduce and test a new measure of semantic ability based on relative lexical diversity. We show how our measure can work alongside existing measures to achieve state of the art correlation with human judgement of quality. We also introduce a new dataset: Rich-Sparse Descriptions, which provides 2K human and machine descriptions to stimulate interest into the semantic evaluation of machine descriptions.
Anthology ID:
W19-0603
Volume:
Proceedings of the 13th International Conference on Computational Semantics - Student Papers
Month:
May
Year:
2019
Address:
Gothenburg, Sweden
Editors:
Simon Dobnik, Stergios Chatzikyriakidis, Vera Demberg, Kathrein Abu Kwaik, Vladislav Maraev
Venue:
IWCS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
15–23
Language:
URL:
https://aclanthology.org/W19-0603
DOI:
10.18653/v1/W19-0603
Bibkey:
Cite (ACL):
Austin Kershaw and Miroslaw Bober. 2019. The Lexical Gap: An Improved Measure of Automated Image Description Quality. In Proceedings of the 13th International Conference on Computational Semantics - Student Papers, pages 15–23, Gothenburg, Sweden. Association for Computational Linguistics.
Cite (Informal):
The Lexical Gap: An Improved Measure of Automated Image Description Quality (Kershaw & Bober, IWCS 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-0603.pdf