Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions

Hannah Craighead, Andrew Caines, Paula Buttery, Helen Yannakoudakis


Abstract
We address the task of automatically grading the language proficiency of spontaneous speech based on textual features from automatic speech recognition transcripts. Motivated by recent advances in multi-task learning, we develop neural networks trained in a multi-task fashion that learn to predict the proficiency level of non-native English speakers by taking advantage of inductive transfer between the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language identification (L1). We encode the transcriptions with both bi-directional recurrent neural networks and with bi-directional representations from transformers, compare against a feature-rich baseline, and analyse performance at different proficiency levels and with transcriptions of varying error rates. Our best performance comes from a transformer encoder with L1 prediction as an auxiliary task. We discuss areas for improvement and potential applications for text-only speech scoring.
Anthology ID:
2020.acl-main.206
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2258–2269
Language:
URL:
https://aclanthology.org/2020.acl-main.206
DOI:
10.18653/v1/2020.acl-main.206
Bibkey:
Cite (ACL):
Hannah Craighead, Andrew Caines, Paula Buttery, and Helen Yannakoudakis. 2020. Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2258–2269, Online. Association for Computational Linguistics.
Cite (Informal):
Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions (Craighead et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.206.pdf
Video:
 http://slideslive.com/38929142