KGEval: Accuracy Estimation of Automatically Constructed Knowledge Graphs

Prakhar Ojha, Partha Talukdar


Abstract
Automatic construction of large knowledge graphs (KG) by mining web-scale text datasets has received considerable attention recently. Estimating accuracy of such automatically constructed KGs is a challenging problem due to their size and diversity. This important problem has largely been ignored in prior research – we fill this gap and propose KGEval. KGEval uses coupling constraints to bind facts and crowdsources those few that can infer large parts of the graph. We demonstrate that the objective optimized by KGEval is submodular and NP-hard, allowing guarantees for our approximation algorithm. Through experiments on real-world datasets, we demonstrate that KGEval best estimates KG accuracy compared to other baselines, while requiring significantly lesser number of human evaluations.
Anthology ID:
D17-1183
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1741–1750
Language:
URL:
https://aclanthology.org/D17-1183
DOI:
10.18653/v1/D17-1183
Bibkey:
Cite (ACL):
Prakhar Ojha and Partha Talukdar. 2017. KGEval: Accuracy Estimation of Automatically Constructed Knowledge Graphs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1741–1750, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
KGEval: Accuracy Estimation of Automatically Constructed Knowledge Graphs (Ojha & Talukdar, EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1183.pdf
Attachment:
 D17-1183.Attachment.zip