Digital Voicing of Silent Speech

David Gaddy, Dan Klein


Abstract
In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected during vocalized speech, we are the first to train from EMG collected during silently articulated speech. We introduce a method of training on silent EMG by transferring audio targets from vocalized to silent signals. Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4% in one data condition and 88% to 68% in another. To spur further development on this task, we share our new dataset of silent and vocalized facial EMG measurements.
Anthology ID:
2020.emnlp-main.445
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5521–5530
Language:
URL:
https://aclanthology.org/2020.emnlp-main.445
DOI:
10.18653/v1/2020.emnlp-main.445
Award:
 Best Paper
Bibkey:
Cite (ACL):
David Gaddy and Dan Klein. 2020. Digital Voicing of Silent Speech. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5521–5530, Online. Association for Computational Linguistics.
Cite (Informal):
Digital Voicing of Silent Speech (Gaddy & Klein, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.445.pdf
Optional supplementary material:
 2020.emnlp-main.445.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38939073
Code
 dgaddy/silent_speech
Data
Silent Speech EMG