Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentation

Ryota Hinami, Shin’ichi Satoh


Abstract
Thanks to the success of object detection technology, we can retrieve objects of the specified classes even from huge image collections. However, the current state-of-the-art object detectors (such as Faster R-CNN) can only handle pre-specified classes. In addition, large amounts of positive and negative visual samples are required for training. In this paper, we address the problem of open-vocabulary object retrieval and localization, where the target object is specified by a textual query (e.g., a word or phrase). We first propose Query-Adaptive R-CNN, a simple extension of Faster R-CNN adapted to open-vocabulary queries, by transforming the text embedding vector into an object classifier and localization regressor. Then, for discriminative training, we then propose negative phrase augmentation (NPA) to mine hard negative samples which are visually similar to the query and at the same time semantically mutually exclusive of the query. The proposed method can retrieve and localize objects specified by a textual query from one million images in only 0.5 seconds with high precision.
Anthology ID:
D18-1281
Volume:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Month:
October-November
Year:
2018
Address:
Brussels, Belgium
Editors:
Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2605–2615
Language:
URL:
https://aclanthology.org/D18-1281
DOI:
10.18653/v1/D18-1281
Bibkey:
Cite (ACL):
Ryota Hinami and Shin’ichi Satoh. 2018. Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2605–2615, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentation (Hinami & Satoh, EMNLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/D18-1281.pdf
Attachment:
 D18-1281.Attachment.pdf
Data
Flickr30K EntitiesVisual Genome