Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer

Chris van der Lee, Bart Verduijn, Emiel Krahmer, Sander Wubben


Abstract
We present an evaluation of PASS, a data-to-text system that generates Dutch soccer reports from match statistics which are automatically tailored towards fans of one club or the other. The evaluation in this paper consists of two studies. An intrinsic human-based evaluation of the system’s output is described in the first study. In this study it was found that compared to human-written texts, computer-generated texts were rated slightly lower on style-related text components (fluency and clarity) and slightly higher in terms of the correctness of given information. Furthermore, results from the first study showed that tailoring was accurately recognized in most cases, and that participants struggled with correctly identifying whether a text was written by a human or computer. The second study investigated if tailoring affects perceived text quality, for which no results were garnered. This lack of results might be due to negative preconceptions about computer-generated texts which were found in the first study.
Anthology ID:
C18-1082
Volume:
Proceedings of the 27th International Conference on Computational Linguistics
Month:
August
Year:
2018
Address:
Santa Fe, New Mexico, USA
Editors:
Emily M. Bender, Leon Derczynski, Pierre Isabelle
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
962–972
Language:
URL:
https://aclanthology.org/C18-1082
DOI:
Bibkey:
Cite (ACL):
Chris van der Lee, Bart Verduijn, Emiel Krahmer, and Sander Wubben. 2018. Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer. In Proceedings of the 27th International Conference on Computational Linguistics, pages 962–972, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Cite (Informal):
Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer (van der Lee et al., COLING 2018)
Copy Citation:
PDF:
https://aclanthology.org/C18-1082.pdf