From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

A new method promises to cut the cost of evaluating large language models by replacing expensive human judges with machine-generated ratings that come with calibrated uncertainty bounds, according to research posted on arXiv [1]. The paper, submitted on 11 June 2026 by Bora Kargi, addresses a persistent bottleneck in artificial intelligence development: ranking new large language models typically requires costly human annotation campaigns at scale [1][2]. LLM-as-a-judge, in which one model scores another, offers a cheaper alternative, but those scores carry systematic errors such as position bias, self-preference, or intransitivity that can miscalibrate rankings [1][2]. The work quantifies judge-human disagreement at two levels. At the local level, the authors propagate calibrated win probabilities rather than hard labels into the Bradley-Terry procedure, estimating per-battle uncertainty from the judge’s own score differences [2]. That step alone brings LLM-derived ratings within 17.9 Elo mean absolute error of human-derived ratings when averaged over 55 held-out models on LMArena [1][2]. At the global level, split conformal prediction is applied to the residual gap between LLM-derived and human-derived Elo ratings, producing prediction intervals with distribution-free marginal coverage guarantees that account for irreducible LLM-human disagreement [2]. Together, the two layers yield a low-cost evaluation tool that provides developers with calibrated Elo estimates and honest uncertainty bounds without access to large-scale human annotations [1][2]. Code has been released on GitHub to facilitate reproducibility [2]. The preprint appeared on arXiv, an open-access repository that hosts e-prints across mathematics, physics, computer science, and related fields and that, as of late 2024, receives about 24,000 submissions per month [6]. The platform also supports community-built tools through its arXivLabs framework, which allows collaborators to develop experimental features such as bibliographic explorers and code finders directly on article pages [4][5].