Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, Curtis Langlotz


Abstract
Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the proposed method to the summarization of radiology reports, where factual correctness is a key requirement. On two separate datasets collected from hospitals, we show via both automatic and human evaluation that the proposed approach substantially improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones.
Anthology ID:
2020.acl-main.458
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5108–5120
Language:
URL:
https://aclanthology.org/2020.acl-main.458
DOI:
10.18653/v1/2020.acl-main.458
Bibkey:
Cite (ACL):
Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. 2020. Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108–5120, Online. Association for Computational Linguistics.
Cite (Informal):
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (Zhang et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.458.pdf
Video:
 http://slideslive.com/38928902
Data
CheXpert