A Re-evaluation of Knowledge Graph Completion Methods

Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, Yiming Yang


Abstract
Knowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. A vast number of state-of-the-art KGC techniques have got published at top conferences in several research fields, including data mining, machine learning, and natural language processing. However, we notice that several recent papers report very high performance, which largely outperforms previous state-of-the-art methods. In this paper, we find that this can be attributed to the inappropriate evaluation protocol used by them and propose a simple evaluation protocol to address this problem. The proposed protocol is robust to handle bias in the model, which can substantially affect the final results. We conduct extensive experiments and report performance of several existing methods using our protocol. The reproducible code has been made publicly available.
Anthology ID:
2020.acl-main.489
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5516–5522
Language:
URL:
https://aclanthology.org/2020.acl-main.489
DOI:
10.18653/v1/2020.acl-main.489
Bibkey:
Cite (ACL):
Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, and Yiming Yang. 2020. A Re-evaluation of Knowledge Graph Completion Methods. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5516–5522, Online. Association for Computational Linguistics.
Cite (Informal):
A Re-evaluation of Knowledge Graph Completion Methods (Sun et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.489.pdf
Video:
 http://slideslive.com/38929034
Code
 svjan5/kg-reeval +  additional community code
Data
FB15kFB15k-237