Lexical Quantile-Based Text Complexity Measure

Maksim Eremeev, Konstantin Vorontsov


Abstract
This paper introduces a new approach to estimating the text document complexity. Common readability indices are based on average length of sentences and words. In contrast to these methods, we propose to count the number of rare words occurring abnormally often in the document. We use the reference corpus of texts and the quantile approach in order to determine what words are rare, and what frequencies are abnormal. We construct a general text complexity model, which can be adjusted for the specific task, and introduce two special models. The experimental design is based on a set of thematically similar pairs of Wikipedia articles, labeled using crowdsourcing. The experiments demonstrate the competitiveness of the proposed approach.
Anthology ID:
R19-1031
Volume:
Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
Month:
September
Year:
2019
Address:
Varna, Bulgaria
Editors:
Ruslan Mitkov, Galia Angelova
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
270–275
Language:
URL:
https://aclanthology.org/R19-1031
DOI:
10.26615/978-954-452-056-4_031
Bibkey:
Cite (ACL):
Maksim Eremeev and Konstantin Vorontsov. 2019. Lexical Quantile-Based Text Complexity Measure. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 270–275, Varna, Bulgaria. INCOMA Ltd..
Cite (Informal):
Lexical Quantile-Based Text Complexity Measure (Eremeev & Vorontsov, RANLP 2019)
Copy Citation:
PDF:
https://aclanthology.org/R19-1031.pdf