The Capability Frontier: Benchmarks Miss 82% of Model Performance

A new study from arXivLabs finds that standard large language model benchmarks miss 82% of collective model performance because they evaluate only a single model on a single run, systematically understating real-world capabilities [1]. The paper introduces the Capability Frontier, a Pareto frontier constructed across multiple models that characterizes the best achievable accuracy at each cost level under optimal selection across models and generations [1]. The authors study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks [2]. The construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples [2]. Correcting for single-model evaluation alone yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with state-of-the-art accuracy matched at an 85% cost reduction [2]. The findings rest on the observation that different models get different questions correct according to their specializations, and that given a budget, multiple generations can be sampled and selectively retained [2]. Controlled probabilistic simulations in the study further show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model [2]. The authors suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings [2]. Benchmark design has long been a contested area in machine learning. The reliance on single-model, single-run reporting echoes earlier debates in other fields about the gap between controlled test conditions and operational performance. For instance, the Sustainable Development Goals framework adopted by United Nations members in 2015 has faced similar measurement challenges: a 2025 UN report noted that only 35% of SDG targets were on track or making moderate progress, with nearly half moving too slowly and 18% in reverse, underscoring how single-point assessments can obscure underlying complexity [6]. The Capability Frontier approach does not require new hardware or model architectures. It reframes evaluation as an optimization problem across existing models and sampling budgets, making it applicable to current deployment pipelines. The paper was submitted on 25 June 2026 and is hosted on arXiv under the artificial intelligence category [1]. The work was developed through arXivLabs, a framework that allows collaborators to build and share new arXiv features while adhering to values of openness, community, excellence, and user data privacy [1].