The Importance of Calibration for Estimating Proportions from Annotations

Dallas Card, Noah A. Smith


Abstract
Estimating label proportions in a target corpus is a type of measurement that is useful for answering certain types of social-scientific questions. While past work has described a number of relevant approaches, nearly all are based on an assumption which we argue is invalid for many problems, particularly when dealing with human annotations. In this paper, we identify and differentiate between two relevant data generating scenarios (intrinsic vs. extrinsic labels), introduce a simple but novel method which emphasizes the importance of calibration, and then analyze and experimentally validate the appropriateness of various methods for each of the two scenarios.
Anthology ID:
N18-1148
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1636–1646
Language:
URL:
https://aclanthology.org/N18-1148
DOI:
10.18653/v1/N18-1148
Bibkey:
Cite (ACL):
Dallas Card and Noah A. Smith. 2018. The Importance of Calibration for Estimating Proportions from Annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1636–1646, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
The Importance of Calibration for Estimating Proportions from Annotations (Card & Smith, NAACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/N18-1148.pdf
Note:
 N18-1148.Notes.pdf