The Benefits of a Model of Annotation

Rebecca J. Passonneau, Bob Carpenter


Abstract
Standard agreement measures for interannotator reliability are neither necessary nor sufficient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained annotators are contrasted with a probabilistic annotation model applied to crowdsourced data. The annotation model provides far more information, including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach.
Anthology ID:
Q14-1025
Erratum e1:
Q14-1025e1
Volume:
Transactions of the Association for Computational Linguistics, Volume 2
Month:
Year:
2014
Address:
Cambridge, MA
Editors:
Dekang Lin, Michael Collins, Lillian Lee
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
311–326
Language:
URL:
https://aclanthology.org/Q14-1025
DOI:
10.1162/tacl_a_00185
Bibkey:
Cite (ACL):
Rebecca J. Passonneau and Bob Carpenter. 2014. The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics, 2:311–326.
Cite (Informal):
The Benefits of a Model of Annotation (Passonneau & Carpenter, TACL 2014)
Copy Citation:
PDF:
https://aclanthology.org/Q14-1025.pdf