Evaluating Factuality in Generation with Dependency-level Entailment

Tanya Goyal, Greg Durrett


Abstract
Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailment systems can be used to identify factual errors; however, these sentence-level entailment models are trained to solve a different problem than generation filtering and they do not localize which part of a generation is non-factual. In this paper, we propose a new formulation of entailment that decomposes it at the level of dependency arcs. Rather than focusing on aggregate decisions, we instead ask whether the semantic relationship manifested by individual dependency arcs in the generated output is supported by the input. Human judgments on this task are difficult to obtain; we therefore propose a method to automatically create data based on existing entailment or paraphrase corpora. Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods or those based on question generation, while additionally localizing the erroneous parts of the generation.
Anthology ID:
2020.findings-emnlp.322
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3592–3603
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.322
DOI:
10.18653/v1/2020.findings-emnlp.322
Bibkey:
Cite (ACL):
Tanya Goyal and Greg Durrett. 2020. Evaluating Factuality in Generation with Dependency-level Entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592–3603, Online. Association for Computational Linguistics.
Cite (Informal):
Evaluating Factuality in Generation with Dependency-level Entailment (Goyal & Durrett, Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.322.pdf
Code
 tagoyal/dae-factuality
Data
MultiNLI