Phonetic and Visual Priors for Decipherment of Informal Romanization

Maria Ryskina, Matthew R. Gormley, Taylor Berg-Kirkpatrick


Abstract
Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common keyboards. Character substitution choices differ between users but have been shown to be governed by the same main principles observed across a variety of languages—namely, character pairs are often associated through phonetic or visual similarity. We propose a noisy-channel WFST cascade model for deciphering the original non-Latin script from observed romanized text in an unsupervised fashion. We train our model directly on romanized data from two languages: Egyptian Arabic and Russian. We demonstrate that adding inductive bias through phonetic and visual priors on character mappings substantially improves the model’s performance on both languages, yielding results much closer to the supervised skyline. Finally, we introduce a new dataset of romanized Russian, collected from a Russian social network website and partially annotated for our experiments.
Anthology ID:
2020.acl-main.737
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8308–8319
Language:
URL:
https://aclanthology.org/2020.acl-main.737
DOI:
10.18653/v1/2020.acl-main.737
Bibkey:
Cite (ACL):
Maria Ryskina, Matthew R. Gormley, and Taylor Berg-Kirkpatrick. 2020. Phonetic and Visual Priors for Decipherment of Informal Romanization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8308–8319, Online. Association for Computational Linguistics.
Cite (Informal):
Phonetic and Visual Priors for Decipherment of Informal Romanization (Ryskina et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.737.pdf
Video:
 http://slideslive.com/38929400
Code
 ryskina/romanization-decipherment