Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision

Aparna Nurani Venkitasubramanian, Tinne Tuytelaars, Marie-Francine Moens


Abstract
We investigate animal recognition models learned from wildlife video documentaries by using the weak supervision of the textual subtitles. This is a particularly challenging setting, since i) the animals occur in their natural habitat and are often largely occluded and ii) subtitles are to a large degree complementary to the visual content, providing a very weak supervisory signal. This is in contrast to most work on integrated vision and language in the literature, where textual descriptions are tightly linked to the image content, and often generated in a curated fashion for the task at hand. In particular, we investigate different image representations and models, including a support vector machine on top of activations of a pretrained convolutional neural network, as well as a Naive Bayes framework on a ‘bag-of-activations’ image representation, where each element of the bag is considered separately. This representation allows key components in the image to be isolated, in spite of largely varying backgrounds and image clutter, without an object detection or image segmentation step. The methods are evaluated based on how well they transfer to unseen camera-trap images captured across diverse topographical regions under different environmental conditions and illumination settings, involving a large domain shift.
Anthology ID:
W17-2003
Volume:
Proceedings of the Sixth Workshop on Vision and Language
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Anya Belz, Erkut Erdem, Katerina Pastra, Krystian Mikolajczyk
Venue:
VL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
21–30
Language:
URL:
https://aclanthology.org/W17-2003
DOI:
10.18653/v1/W17-2003
Bibkey:
Cite (ACL):
Aparna Nurani Venkitasubramanian, Tinne Tuytelaars, and Marie-Francine Moens. 2017. Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision. In Proceedings of the Sixth Workshop on Vision and Language, pages 21–30, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Learning to Recognize Animals by Watching Documentaries: Using Subtitles as Weak Supervision (Nurani Venkitasubramanian et al., VL 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-2003.pdf
Data
ImageNet