On-The-Fly Information Retrieval Augmentation for Language Models

Hai Wang, David McAllester


Abstract
Here we experiment with the use of information retrieval as an augmentation for pre-trained language models. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.
Anthology ID:
2020.nuse-1.14
Volume:
Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events
Month:
July
Year:
2020
Address:
Online
Editors:
Claire Bonial, Tommaso Caselli, Snigdha Chaturvedi, Elizabeth Clark, Ruihong Huang, Mohit Iyyer, Alejandro Jaimes, Heng Ji, Lara J. Martin, Ben Miller, Teruko Mitamura, Nanyun Peng, Joel Tetreault
Venues:
NUSE | WNU
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
114–119
Language:
URL:
https://aclanthology.org/2020.nuse-1.14
DOI:
10.18653/v1/2020.nuse-1.14
Bibkey:
Cite (ACL):
Hai Wang and David McAllester. 2020. On-The-Fly Information Retrieval Augmentation for Language Models. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 114–119, Online. Association for Computational Linguistics.
Cite (Informal):
On-The-Fly Information Retrieval Augmentation for Language Models (Wang & McAllester, NUSE-WNU 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.nuse-1.14.pdf
Video:
 http://slideslive.com/38929754
Data
ECB+