When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking

A new meta-analysis argues that standard Knowledge Graph Completion benchmarks produce contradictory model rankings, undermining the reliability of head-to-head comparisons. The study, submitted to arXiv on 9 Jun 2026, reframes evaluation as a multi-criteria decision-making problem to resolve these inconsistencies [1][2]. The research identifies a persistent fragmentation in how KGC models are assessed. Standard metrics such as Mean Reciprocal Rank (MRR), Hits@$k$, and Mean Rank often yield conflicting orderings across datasets. A model that leads on MRR may trail on Hits@1, and strong performance on one dataset may not generalize to another [1][2]. This fragmentation hinders comparison, enables selective reporting, and obscures real progress, according to the paper [2]. To address this, the authors conducted a meta-analysis of seven aggregators across five tests: consistency, cross-dataset stability, metric independence, robustness under noise, and generalizability. Each test was averaged over leave-one-model-out and leave-one-group-out removals so that reliability reflects aggregator behavior across diverse model subsets [2]. A Pareto-optimal analysis identified Z-score as the most balanced aggregator. Under this framework, DualE ranked highest for tail prediction, while Flow-Modulated Scoring (FMS) ranked highest for relation prediction [1][2]. The study also includes a test-sensitivity analysis using the same removal protocols. It found that consistency and stability are largely removal-invariant, whereas generalizability and independence are the most sensitive to which models are included or excluded [2]. The framework offers evidence-based guidance for aggregator selection and model benchmarking in KGC [1][2]. The paper was posted on arXiv, an open-access repository for electronic preprints that has hosted over two million articles since its launch in 1991 [6]. arXiv itself does not conduct peer review, but it provides a platform for rapid dissemination of research findings at no cost to readers and submitters [4][6]. The repository has also introduced arXivLabs, a framework for community-developed tools such as bibliographic explorers and code finders that appear on article record pages [4][5].