Facet-Aware Evaluation for Extractive Summarization

Yuning Mao, Liyuan Liu, Qi Zhu, Xiang Ren, Jiawei Han


Abstract
Commonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage in extracted summaries. Specifically, we treat each sentence in the reference summary as a facet, identify the sentences in the document that express the semantics of each facet as support sentences of the facet, and automatically evaluate extractive summarization methods by comparing the indices of extracted sentences and support sentences of all the facets in the reference summary. To facilitate this new evaluation setup, we construct an extractive version of the CNN/Daily Mail dataset and perform a thorough quantitative investigation, through which we demonstrate that facet-aware evaluation manifests better correlation with human judgment than ROUGE, enables fine-grained evaluation as well as comparative analysis, and reveals valuable insights of state-of-the-art summarization methods. Data can be found at https://github.com/morningmoni/FAR.
Anthology ID:
2020.acl-main.445
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4941–4957
Language:
URL:
https://aclanthology.org/2020.acl-main.445
DOI:
10.18653/v1/2020.acl-main.445
Bibkey:
Cite (ACL):
Yuning Mao, Liyuan Liu, Qi Zhu, Xiang Ren, and Jiawei Han. 2020. Facet-Aware Evaluation for Extractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4941–4957, Online. Association for Computational Linguistics.
Cite (Informal):
Facet-Aware Evaluation for Extractive Summarization (Mao et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.445.pdf
Video:
 http://slideslive.com/38928688
Code
 morningmoni/FAR
Data
CNN/Daily Mail