UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation

Jian Guan, Minlie Huang


Abstract
Despite the success of existing referenced metrics (e.g., BLEU and MoverScore), they correlate poorly with human judgments for open-ended text generation including story or dialog generation because of the notorious one-to-many issue: there are many plausible outputs for the same input, which may differ substantially in literal or semantics from the limited number of given references. To alleviate this issue, we propose UNION, a learnable UNreferenced metrIc for evaluating Open-eNded story generation, which measures the quality of a generated story without any reference. Built on top of BERT, UNION is trained to distinguish human-written stories from negative samples and recover the perturbation in negative stories. We propose an approach of constructing negative samples by mimicking the errors commonly observed in existing NLG models, including repeated plots, conflicting logic, and long-range incoherence. Experiments on two story datasets demonstrate that UNION is a reliable measure for evaluating the quality of generated stories, which correlates better with human judgments and is more generalizable than existing state-of-the-art metrics.
Anthology ID:
2020.emnlp-main.736
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9157–9166
Language:
URL:
https://aclanthology.org/2020.emnlp-main.736
DOI:
10.18653/v1/2020.emnlp-main.736
Bibkey:
Cite (ACL):
Jian Guan and Minlie Huang. 2020. UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9157–9166, Online. Association for Computational Linguistics.
Cite (Informal):
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (Guan & Huang, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.736.pdf
Optional supplementary material:
 2020.emnlp-main.736.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38938672
Code
 thu-coai/UNION
Data
ROCStoriesWritingPrompts