Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers

Georgios Spithourakis, Sebastian Riedel


Abstract
Numeracy is the ability to understand and work with numbers. It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains. In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composition, and propose a novel neural architecture that uses a continuous probability density function to model numerals from an open vocabulary. Our evaluation on clinical and scientific datasets shows that using hierarchical models to distinguish numerals from words improves a perplexity metric on the subset of numerals by 2 and 4 orders of magnitude, respectively, over non-hierarchical models. A combination of strategies can further improve perplexity. Our continuous probability density function model reduces mean absolute percentage errors by 18% and 54% in comparison to the second best strategy for each dataset, respectively.
Anthology ID:
P18-1196
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2104–2115
Language:
URL:
https://aclanthology.org/P18-1196
DOI:
10.18653/v1/P18-1196
Bibkey:
Cite (ACL):
Georgios Spithourakis and Sebastian Riedel. 2018. Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2104–2115, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers (Spithourakis & Riedel, ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-1196.pdf
Presentation:
 P18-1196.Presentation.pdf
Video:
 https://aclanthology.org/P18-1196.mp4