Transparent text quality assessment with convolutional neural networks

Robert Östling, Gintare Grigonyte


Abstract
We present a very simple model for text quality assessment based on a deep convolutional neural network, where the only supervision required is one corpus of user-generated text of varying quality, and one contrasting text corpus of consistently high quality. Our model is able to provide local quality assessments in different parts of a text, which allows visual feedback about where potentially problematic parts of the text are located, as well as a way to evaluate which textual features are captured by our model. We evaluate our method on two corpora: a large corpus of manually graded student essays and a longitudinal corpus of language learner written production, and find that the text quality metric learned by our model is a fairly strong predictor of both essay grade and learner proficiency level.
Anthology ID:
W17-5031
Volume:
Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Joel Tetreault, Jill Burstein, Claudia Leacock, Helen Yannakoudakis
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
282–286
Language:
URL:
https://aclanthology.org/W17-5031
DOI:
10.18653/v1/W17-5031
Bibkey:
Cite (ACL):
Robert Östling and Gintare Grigonyte. 2017. Transparent text quality assessment with convolutional neural networks. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 282–286, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Transparent text quality assessment with convolutional neural networks (Östling & Grigonyte, BEA 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-5031.pdf