Deployment-complete benchmarking

A new framework called deployment-complete benchmarking tests whether benchmark scores actually determine real-world deployment decisions, according to a paper posted to arXiv on 25 May 2026 [1]. The authors find that high benchmark performance often masks critical gaps when models face unmeasured conditions [2]. The study, authored by El Mustapha Mansouri, introduces a formal test: a benchmark is complete for a claim only when the deployment action remains constant across every evidence fiber — the set of models that produce the same benchmark result [2]. When fibers are mixed, meaning identical scores lead to different deployment choices, the benchmark is missing information essential to the decision [2]. In controlled experiments, benchmark-channel conformal coverage reached 94.98%, but that reliability collapsed to 10.07% when transferred to an unmeasured deployment channel [1]. By contrast, response-rank intervals achieved 94.91% coverage, suggesting that ranking-based approaches can preserve uncertainty better than score-based methods [2]. Even when benchmark error was zero, only 45.4% of candidates could be certified at the largest residual size, underscoring that perfect benchmark accuracy does not guarantee deployment readiness [1]. Public audits of widely used benchmarks revealed widespread incompleteness. In the Tox21 toxicology benchmark, 97.9% of fibers were mixed, meaning the same score frequently corresponded to conflicting deployment actions [2]. Audits of the main Matbench and JARVIS materials-science benchmarks found a median certifiable fraction of zero, indicating that no deployment decision could be reliably supported by the benchmark evidence alone [1]. The paper also proposes a “certify-then-acquire” protocol. In held-out replays, this approach reduced false decisions from 1.19% to 0.027% in Tox21 and from 20.3% to 0.128% in JARVIS [1]. The protocol also changed which model was selected and identified probes that were relevant to deployment, rather than simply optimizing for benchmark scores [2]. Benchmark evaluations are already central to how large language models and other AI systems are assessed for reasoning, factual accuracy, and safety [3]. The new work argues that reporting a score alone is insufficient; deployment-ready benchmarks should instead disclose the evidence, the actions they support, the remaining ambiguity, and the cost of achieving completeness [2].