Extrinsic Corpus Evaluation with a Collocation Dictionary Task

Adam Kilgarriff, Pavel Rychlý, Miloš Jakubíček, Vojtěch Kovář, Vít Baisa, Lucia Kocincová


Abstract
The NLP researcher or application-builder often wonders “what corpus should I use, or should I build one of my own? If I build one of my own, how will I know if I have done a good job?” Currently there is very little help available for them. They are in need of a framework for evaluating corpora. We develop such a framework, in relation to corpora which aim for good coverage of ‘general language’. The task we set is automatic creation of a publication-quality collocations dictionary. For a sample of 100 headwords of Czech and 100 of English, we identify a gold standard dataset of (ideally) all the collocations that should appear for these headwords in such a dictionary. The datasets are being made available alongside this paper. We then use them to determine precision and recall for a range of corpora, with a range of parameters.
Anthology ID:
L14-1431
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
545–552
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/52_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Adam Kilgarriff, Pavel Rychlý, Miloš Jakubíček, Vojtěch Kovář, Vít Baisa, and Lucia Kocincová. 2014. Extrinsic Corpus Evaluation with a Collocation Dictionary Task. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 545–552, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
Extrinsic Corpus Evaluation with a Collocation Dictionary Task (Kilgarriff et al., LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/52_Paper.pdf