Evaluating Text GANs as Language Models

Guy Tevet, Gavriel Habib, Vered Shwartz, Jonathan Berant


Abstract
Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.
Anthology ID:
N19-1233
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2241–2247
Language:
URL:
https://aclanthology.org/N19-1233
DOI:
10.18653/v1/N19-1233
Bibkey:
Cite (ACL):
Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. 2019. Evaluating Text GANs as Language Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2241–2247, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Evaluating Text GANs as Language Models (Tevet et al., NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1233.pdf
Code
 GuyTevet/SeqGAN-eval