Language Richness of the Web

Martin Majliš, Zdeněk Žabokrtský


Abstract
We have built a corpus containing texts in 106 languages from texts available on the Internet and on Wikipedia. The W2C Web Corpus contains 54.7~GB of text and the W2C Wiki Corpus contains 8.5~GB of text. The W2C Web Corpus contains more than 100~MB of text available for 75 languages. At least 10~MB of text is available for 100 languages. These corpora are a unique data source for linguists, since they outclass all published works both in the size of the material collected and the number of languages covered. This language data resource can be of use particularly to researchers specialized in multilingual technologies development. We also developed software that greatly simplifies the creation of a new text corpus for a given language, using text materials freely available on the Internet. Special attention was given to components for filtering and de-duplication that allow to keep the material quality very high.
Anthology ID:
L12-1110
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
2927–2934
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/267_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Martin Majliš and Zdeněk Žabokrtský. 2012. Language Richness of the Web. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2927–2934, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
Language Richness of the Web (Majliš & Žabokrtský, LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/267_Paper.pdf