Annotating omission in statement pairs

Héctor Martínez Alonso, Amaury Delamaire, Benoît Sagot


Abstract
We focus on the identification of omission in statement pairs. We compare three annotation schemes, namely two different crowdsourcing schemes and manual expert annotation. We show that the simplest of the two crowdsourcing approaches yields a better annotation quality than the more complex one. We use a dedicated classifier to assess whether the annotators’ behavior can be explained by straightforward linguistic features. The classifier benefits from a modeling that uses lexical information beyond length and overlap measures. However, for our task, we argue that expert and not crowdsourcing-based annotation is the best compromise between annotation cost and quality.
Anthology ID:
W17-0805
Volume:
Proceedings of the 11th Linguistic Annotation Workshop
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Nathan Schneider, Nianwen Xue
Venue:
LAW
SIG:
SIGANN
Publisher:
Association for Computational Linguistics
Note:
Pages:
41–45
Language:
URL:
https://aclanthology.org/W17-0805
DOI:
10.18653/v1/W17-0805
Bibkey:
Cite (ACL):
Héctor Martínez Alonso, Amaury Delamaire, and Benoît Sagot. 2017. Annotating omission in statement pairs. In Proceedings of the 11th Linguistic Annotation Workshop, pages 41–45, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Annotating omission in statement pairs (Martínez Alonso et al., LAW 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-0805.pdf
Code
 hectormartinez/verdidata