Knowledge Base Completion: Baselines Strike Back

Many papers have been published on the knowledge base completion task in the past few years. Most of these introduce novel architectures for relation learning that are evaluated on standard datasets like FB15k and WN18. This paper shows that the accuracy of almost all models published on the FB15k can be outperformed by an appropriately tuned baseline — our reimplementation of the DistMult model. Our findings cast doubt on the claim that the performance improvements of recent models are due to architectural changes as opposed to hyper-parameter tuning or different training objectives. This should prompt future research to re-consider how the performance of models is evaluated and reported.


Introduction
Projects such as Wikidata 1 or earlier Freebase (Bollacker et al., 2008) have successfully accumulated a formidable amount of knowledge in the form of entity1 -relation -entity2 triplets. Given this vast body of knowledge, it would be extremely useful to teach machines to reason over such knowledge bases. One possible way to test such reasoning is knowledge base completion (KBC).
The goal of the KBC task is to fill in the missing piece of information into an incomplete triple. For instance, given a query Donald Trump, president of, ? one should predict that the target entity is USA.
More formally, given a set of entities E and a set of binary relations R over these entities, a knowledge base (sometimes also referred to as a knowl-edge graph) can be specified by a set of triplets h, r, t where h, t ∈ E are head and tail entities respectively and r ∈ R is a relation between them. In entity KBC the task is to predict either the tail entity given a query h, r, ? , or to predict the head entity given ?, r, t . Not only can this task be useful to test the generic ability of a system to reason over a knowledge base, but it can also find use in expanding existing incomplete knowledge bases by deducing new entries from existing ones.
An extensive amount of work has been published on this task (for a review see (Nickel et al., 2015;Nguyen, 2017), for a plain list of citations see Table 2). Among those DistMult  is one of the simplest. 2 Still this paper shows that even a simple model with proper hyperparameters and training objective evaluated using the standard metric of Hits@10 can outperform 27 out of 29 models which were evaluated on two standard KBC datasets, WN18 and FB15k (Bordes et al., 2013).
This suggests that there may be a huge space for improvement in hyper-parameter tuning even for the more complex models, which may be in many ways better suited for relational learning, e.g. can capture directed relations.

The Model
Inspired by the success of word embeddings in natural language processing, distributional models for KBC have recently been extensively studied. Distributional models represent the entities and sometimes even the relations as N -dimensional real vectors 3 , we will denote these vectors by bold font, h, r, t ∈ R N . The DistMult model was introduced by . Subsequently Toutanova and Chen (2015) achieved better empirical results with the same model by changing hyper-parameters of the training procedure and by using negative-log likelihood of softmax instead of L1-based max-margin ranking loss. Trouillon et al. (2016) obtained even better empirical result on the FB15k dataset just by changing DistMult's hyper-parameters.
DistMult model computes a score for each triplet h, r, t as where W r is a diagonal matrix with elements of vector r on its diagonal. Therefore the model can be alternatively rewritten as shown in the second equality.
In the end our implementation normalizes the scores by a softmax function. That is where E h,r is a set of candidate answer entities for the h, r, ? query.

Experiments
Datasets. In our experiments we use two standard datasets WN18 derived from WordNet (Fellbaum, 1998) and FB15k derived from the Freebase knowledge graph (Bollacker et al., 2008).
Method. For evaluation, we use the filtered evaluation protocol proposed by Bordes et al. (2013). During training and validation we transform each triplet h, r, t into two examples: tail query h, r, ? and head query ?, r, t . We train the model by minimizing negative log-likelihood (NLL) of the ground truth triplet h, r, t against randomly sampled pool of M negative triplets h, r, t , t ∈ E \ {t} (this applies for tail queries, head queries are handled analogically).
In the filtered protocol we rank the validation or test set triplet against all corrupted (supposedly untrue) triplets -those that do not appear in the train, valid and test dataset (excluding the test set triplet in question itself). Formally, for a query h, r, ? where the correct answer is t, we compute the rank of h, r, t in a candidate set C h,r = { h, r, t : ∀t ∈ E} \ (T rain ∪ V alid ∪ T est) ∪ { h, r, t }, where T rain, V alid and T est are sets of true triplets. Head queries ?, r, t are handled analogically. Note that softmax normalization is suitable under the filtered protocol since exactly one correct triplet is guaranteed to be among the candidates.
In our preliminary experiments on FB15k, we varied the batch size b, embedding dimensionality N , number of negative samples in training M , L2 regularization parameter and learning rate lr. Based on these experiments we fixed lr=0.001, L2=0.0 and we decided to focus on influence of batch size, embedding dimension and number of negative samples. For final experiments we trained several models from hyperparameter range: N ∈ {128, 256, 512, 1024}, b ∈ {16, 32, 64, 128, 256, 512, 1024, 2048} and M ∈ {20, 50, 200, 500, 1000, 2000}. We train the final models using Adam (Kingma and Ba, 2015) optimizer (lr = 0.001, β 1 = 0.9, β 2 = 0.999, = 10 −8 , decay = 0.0). We also performed limited experiments with Adagrad, Adadelta and plain SGD. Adagrad usually required substantially more iterations than ADAM to achieve the same performance. We failed to obtain competitive performance with Adadelta and plain SGD. On FB15k and WN18 validation datasets the best hyper-parameter combinations were N = 512, b = 2048, M = 2000 and N = 256, b = 1024, M = 1000, respectively. Note that we tried substantially more hyperparameter combinations on FB15k than on WN18. Unlike most previous works we do not normalize neither entity nor relation embeddings.
To prevent over-fitting, we stop training once Hits@10 stop improving on the validation set. On the FB15k dataset our Keras (Chollet, 2015) based implementation with TensorFlow (Abadi et al., 2015) backend needed about 4 hours to converge when run on a single GeForce GTX 1080 GPU.
Results. Besides single models, we also evaluated performance of a simple ensemble that averages predictions of multiple models. This technique consistently improves performance of machine learning models in many domains and it slightly improved results also in this case.
The results of our experiments together with previous results from the literature are shown in Table 2. DistMult with proper hyperparameters twice achieves the second best score and once the third best score in three out of four commonly reported benchmarks (mean rank (MR) and Hits@10 on WN18 and FB15k). On FB15k only the IRN model (Shen et al., 2016) shows better Hits@10 and the ProjE (Shi and Weniger, 2017) has better MR.
Our implementation has the best reported mean reciprocal rank (MRR) on FB15k, however this metric is not reported that often. MRR is a metric of ranking quality that is less sensitive to outliers than MR.
On WN18 dataset again the IRN model together with R-GCN+ shows better Hits@10. However, in MR and MRR DistMult performs poorly. Even though DistMult's inability to model asymmetric relations still allows it to achieve competitive results in Hits@10 the other metrics clearly show its limitations. These results highlight qualitative differences between FB15k and WN18 datasets.
Interestingly on FB15k recently published models (including our baseline) that use only r and h or t as their input outperform models that utilize richer features such as text or knowledge base path information. This shows a possible gap for future improvement. Table 1 shows accuracy (Hits@1) of several models that reported this metric. On WN18 our implementation performs worse than HolE and ComplEx models (that are equivalent as shown by Hayashi and Shimbo (2017)). On FB15k our implementation outperforms all other models.

Hyper-parameter influence on FB15k
In our experiments on FB15k we found that increasing the number of negative examples M had a positive effect on performance.
Another interesting observation is that batch size has a strong influence on final performance. Larger batch size always lead to better results, for instance Hits@10 improved by 14.2% absolute when the batch size was increased from 16 to 2048. See Figure 1 for details.
Compared to previous works that trained Dist-Mult on these datasets (for results see bottom of Table 2) we use different training objective than  and Trouillon et al. (2017) that optimized max margin objective and NLL of softplus activation function (sof tplus(x) = ln(1 + e x )), respectively. Similarly to Toutanova and Chen (2015) we use NLL of softmax function, however we use ADAM optimizer instead of RProp (Riedmiller and Braun, 1993).   (Nickel et al., 2016), (Trouillon et al., 2017) and (Schlichtkrull et al., 2017), respectively. Our implementation is listed in the last row.

Conclusion
Simple conclusions from our work are: 1) Increasing batch size dramatically improves performance of DistMult, which raises a question whether other models would also significantly benefit from similar hyper-parameter tuning or different training objectives; 2) In the future it might be better to focus more on metrics less frequently used in this domain, like Hits@1 (accuracy) and MRR since for instance on WN18 many models achieve similar, very high Hits@10, however even models that are competitive in Hits@10 underperform in Hits@1, which is the case of our DistMult implementation.
A lot of research focus has recently been centred on the filtered scenario which is why we decided to use it in this study. An advantage is that it is easy to evaluate. However the scenario trains the model to expect that there is only a single correct answer among the candidates which is unrealistic in the context of knowledge bases. Hence

None
Unstructured (Bordes et al., 2014) 304 38.2 -979 6.3 -TransE (Bordes et al., 2013) 251 89.2 -125 47.1 -TransH (Wang et al., 2014) 303 86.7 -87 64.4 -TransR (Lin et al., 2015b) 225 92.0 -77 68.7 -CTransR (Lin et al., 2015b) 218 92.3 -75 70.2 -KG2E  331 92.8 -59 74.0 -TransD  212 92.  Table 2: Entity prediction results. MR, H10 and MRR denote evaluation metrics of mean rank, Hits@10 (in %) and mean reciprocal rank, respectively. The three best results for each metric are in bold. Additionally the best result is underlined. The first group (above the first double line) lists models that were trained only on the knowledge base and they do not use any additional input besides the source entity and the relation. The second group shows models that use path information, e.g. they consider paths between source and target entities as additional features. The models from the third group were trained with additional textual data. In the last group we list various implementations of the DistMult model including our implementation on the last two lines. Since DistMult does not use any additional features these results should be compared to the models from the first group. "NLFeat" abbreviates Node+LinkFeat model from (Toutanova and Chen, 2015). The results for NTN (Socher et al., 2013) listed in this table are taken from . This table was adapted from (Nguyen, 2017). future research could focus more on the raw scenario which however requires using other information retrieval metrics such as mean average precision (MAP), previously used in KBC for instance by Das et al. (2017).
We see this preliminary work as a small contribution to the ongoing discussion in the machine learning community about the current strong focus on state-of-the-art empirical results when it might be sometimes questionable whether they were achieved due to a better model/algorithm or just by more extensive hyper-parameter search. For broader discussion see (Church, 2017).
In light of these results we think that the field would benefit from a large-scale empirical comparative study of different KBC algorithms, similar to a recent study of word embedding models (Levy et al., 2015).