Building a Japanese Typo Dataset from Wikipedia’s Revision History

Yu Tanaka, Yugo Murawaki, Daisuke Kawahara, Sadao Kurohashi


Abstract
User generated texts contain many typos for which correction is necessary for NLP systems to work. Although a large number of typo–correction pairs are needed to develop a data-driven typo correction system, no such dataset is available for Japanese. In this paper, we extract over half a million Japanese typo–correction pairs from Wikipedia’s revision history. Unlike other languages, Japanese poses unique challenges: (1) Japanese texts are unsegmented so that we cannot simply apply a spelling checker, and (2) the way people inputting kanji logographs results in typos with drastically different surface forms from correct ones. We address them by combining character-based extraction rules, morphological analyzers to guess readings, and various filtering methods. We evaluate the dataset using crowdsourcing and run a baseline seq2seq model for typo correction.
Anthology ID:
2020.acl-srw.31
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
Month:
July
Year:
2020
Address:
Online
Editors:
Shruti Rijhwani, Jiangming Liu, Yizhong Wang, Rotem Dror
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
230–236
Language:
URL:
https://aclanthology.org/2020.acl-srw.31
DOI:
10.18653/v1/2020.acl-srw.31
Bibkey:
Cite (ACL):
Yu Tanaka, Yugo Murawaki, Daisuke Kawahara, and Sadao Kurohashi. 2020. Building a Japanese Typo Dataset from Wikipedia’s Revision History. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 230–236, Online. Association for Computational Linguistics.
Cite (Informal):
Building a Japanese Typo Dataset from Wikipedia’s Revision History (Tanaka et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-srw.31.pdf
Video:
 http://slideslive.com/38928676