Distilling Neural Networks for Greener and Faster Dependency Parsing

Mark Anderson, Carlos Gómez-Rodríguez


Abstract
The carbon footprint of natural language processing research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation to improve the efficiency of the Biaffine dependency parser which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat and Manning, 2017). When distilling to 20% of the original model’s trainable parameters, we only observe an average decrease of ∼1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.30x (1.19x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80% for some treebanks. Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.
Anthology ID:
2020.iwpt-1.2
Volume:
Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies
Month:
July
Year:
2020
Address:
Online
Editors:
Gosse Bouma, Yuji Matsumoto, Stephan Oepen, Kenji Sagae, Djamé Seddah, Weiwei Sun, Anders Søgaard, Reut Tsarfaty, Dan Zeman
Venue:
IWPT
SIG:
SIGPARSE
Publisher:
Association for Computational Linguistics
Note:
Pages:
2–13
Language:
URL:
https://aclanthology.org/2020.iwpt-1.2
DOI:
10.18653/v1/2020.iwpt-1.2
Bibkey:
Cite (ACL):
Mark Anderson and Carlos Gómez-Rodríguez. 2020. Distilling Neural Networks for Greener and Faster Dependency Parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 2–13, Online. Association for Computational Linguistics.
Cite (Informal):
Distilling Neural Networks for Greener and Faster Dependency Parsing (Anderson & Gómez-Rodríguez, IWPT 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.iwpt-1.2.pdf
Video:
 http://slideslive.com/38929669
Data
Penn Treebank