Comparison of Diverse Decoding Methods from Conditional Language Models

Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, Chris Callison-Burch


Abstract
While conditional language models have greatly improved in their ability to output high quality natural language, many NLP applications benefit from being able to generate a diverse set of candidate sequences. Diverse decoding strategies aim to, within a given-sized candidate list, cover as much of the space of high-quality outputs as possible, leading to improvements for tasks that rerank and combine candidate outputs. Standard decoding methods, such as beam search, optimize for generating high likelihood sequences rather than diverse ones, though recent work has focused on increasing diversity in these methods. In this work, we perform an extensive survey of decoding-time strategies for generating diverse outputs from a conditional language model. In addition, we present a novel method where we over-sample candidates, then use clustering to remove similar sequences, thus achieving high diversity without sacrificing quality.
Anthology ID:
P19-1365
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Anna Korhonen, David Traum, Lluís Màrquez
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3752–3762
Language:
URL:
https://aclanthology.org/P19-1365
DOI:
10.18653/v1/P19-1365
Bibkey:
Cite (ACL):
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of Diverse Decoding Methods from Conditional Language Models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752–3762, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Comparison of Diverse Decoding Methods from Conditional Language Models (Ippolito et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1365.pdf
Code
 rekriz11/DeDiv