It’s Commonsense, isn’t it? Demystifying Human Evaluations in Commonsense-Enhanced NLG Systems

Miruna-Adriana Clinciu, Dimitra Gkatzia, Saad Mahamood


Abstract
Common sense is an integral part of human cognition which allows us to make sound decisions, communicate effectively with others and interpret situations and utterances. Endowing AI systems with commonsense knowledge capabilities will help us get closer to creating systems that exhibit human intelligence. Recent efforts in Natural Language Generation (NLG) have focused on incorporating commonsense knowledge through large-scale pre-trained language models or by incorporating external knowledge bases. Such systems exhibit reasoning capabilities without common sense being explicitly encoded in the training set. These systems require careful evaluation, as they incorporate additional resources during training which adds additional sources of errors. Additionally, human evaluation of such systems can have significant variation, making it impossible to compare different systems and define baselines. This paper aims to demystify human evaluations of commonsense-enhanced NLG systems by proposing the Commonsense Evaluation Card (CEC), a set of recommendations for evaluation reporting of commonsense-enhanced NLG systems, underpinned by an extensive analysis of human evaluations reported in the recent literature.
Anthology ID:
2021.humeval-1.1
Volume:
Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)
Month:
April
Year:
2021
Address:
Online
Editors:
Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, Anastasia Shimorina
Venue:
HumEval
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–12
Language:
URL:
https://aclanthology.org/2021.humeval-1.1
DOI:
Bibkey:
Cite (ACL):
Miruna-Adriana Clinciu, Dimitra Gkatzia, and Saad Mahamood. 2021. It’s Commonsense, isn’t it? Demystifying Human Evaluations in Commonsense-Enhanced NLG Systems. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 1–12, Online. Association for Computational Linguistics.
Cite (Informal):
It’s Commonsense, isn’t it? Demystifying Human Evaluations in Commonsense-Enhanced NLG Systems (Clinciu et al., HumEval 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.humeval-1.1.pdf
Video:
 https://www.youtube.com/watch?v=LlrsKZOKIoo
Video:
 https://aclanthology.org/2021.humeval-1.1.mp4
Data
ConceptNet