C4Corpus: Multilingual Web-size Corpus with Free License

Ivan Habernal, Omnia Zayed, Iryna Gurevych


Abstract
Large Web corpora containing full documents with permissive licenses are crucial for many NLP tasks. In this article we present the construction of 12 million-pages Web corpus (over 10 billion tokens) licensed under CreativeCommons license family in 50+ languages that has been extracted from CommonCrawl, the largest publicly available general Web crawl to date with about 2 billion crawled URLs. Our highly-scalable Hadoop-based framework is able to process the full CommonCrawl corpus on 2000+ CPU cluster on the Amazon Elastic Map/Reduce infrastructure. The processing pipeline includes license identification, state-of-the-art boilerplate removal, exact duplicate and near-duplicate document removal, and language detection. The construction of the corpus is highly configurable and fully reproducible, and we provide both the framework (DKPro C4CorpusTools) and the resulting data (C4Corpus) to the research community.
Anthology ID:
L16-1146
Volume:
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
Month:
May
Year:
2016
Address:
Portorož, Slovenia
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
914–922
Language:
URL:
https://aclanthology.org/L16-1146
DOI:
Bibkey:
Cite (ACL):
Ivan Habernal, Omnia Zayed, and Iryna Gurevych. 2016. C4Corpus: Multilingual Web-size Corpus with Free License. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 914–922, Portorož, Slovenia. European Language Resources Association (ELRA).
Cite (Informal):
C4Corpus: Multilingual Web-size Corpus with Free License (Habernal et al., LREC 2016)
Copy Citation:
PDF:
https://aclanthology.org/L16-1146.pdf
Code
 dkpro/dkpro-c4corpus