Treat the system like a human student: Automatic naturalness evaluation of generated text without reference texts

Isabel Groves, Ye Tian, Ioannis Douratsos


Abstract
The current most popular method for automatic Natural Language Generation (NLG) evaluation is comparing generated text with human-written reference sentences using a metrics system, which has drawbacks around reliability and scalability. We draw inspiration from second language (L2) assessment and extract a set of linguistic features to predict human judgments of sentence naturalness. Our experiment using a small dataset showed that the feature-based approach yields promising results, with the added potential of providing interpretability into the source of the problems.
Anthology ID:
W18-6512
Volume:
Proceedings of the 11th International Conference on Natural Language Generation
Month:
November
Year:
2018
Address:
Tilburg University, The Netherlands
Editors:
Emiel Krahmer, Albert Gatt, Martijn Goudbeek
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
109–118
Language:
URL:
https://aclanthology.org/W18-6512
DOI:
10.18653/v1/W18-6512
Bibkey:
Cite (ACL):
Isabel Groves, Ye Tian, and Ioannis Douratsos. 2018. Treat the system like a human student: Automatic naturalness evaluation of generated text without reference texts. In Proceedings of the 11th International Conference on Natural Language Generation, pages 109–118, Tilburg University, The Netherlands. Association for Computational Linguistics.
Cite (Informal):
Treat the system like a human student: Automatic naturalness evaluation of generated text without reference texts (Groves et al., INLG 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-6512.pdf