Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

A new statistical framework proposes to bring uncertainty quantification to multi-task model leaderboards, addressing a gap in how pretrained models are ranked across diverse benchmarks. The method, posted on arXiv, constructs rank intervals with formal guarantees at both the individual task level and the aggregate leaderboard level [1]. Current leaderboard aggregation methods typically collapse model performance across tasks into a single ranking without accounting for the variability in how a model performs from one task to the next [1]. While some recent work has introduced interval-based rankings, a principled way to propagate task-level uncertainty into an overall leaderboard rank has been missing [2]. The new hierarchical framework tackles this by building task-level rank confidence intervals from pairwise model comparisons, then using a conformal prediction approach to create leaderboard-level rank prediction intervals [1]. The authors state this allows for reliable rank quantification both for tasks a model has already been evaluated on and for new, unseen tasks [2]. The paper’s experiments, conducted on simulated data and the TabArena and PromptEval benchmarks, which is based on the MMLU dataset, show the method produces intervals that are statistically valid and informative [1]. The work arrives as the volume of machine learning research on the preprint server continues to grow. arXiv, which was founded in 1991, now receives roughly 24,000 new articles per month as of late 2024 [6]. The platform hosts papers across physics, computer science, and other quantitative fields, and is not peer-reviewed before posting [6]. The article appears with the standard arXivLabs integration, a framework launched in 2020 that allows community collaborators to build experimental tools directly on the site [5]. These tools, which appear as tabs on an article’s abstract page, include citation explorers and code-finding services [4]. arXivLabs partners must adhere to the repository’s values of openness, community, excellence, and user data privacy, and are limited to accessing minimal, anonymized user data [5]. The framework is currently on a hiatus for new proposals while the arXiv development team focuses on migrating its systems to the cloud [3]. Large language models, which are a primary subject of the leaderboards the paper aims to improve, are typically evaluated on benchmarks that measure reasoning, factual accuracy, and safety [8]. The proposed rank-interval method offers a new statistical tool for interpreting those evaluations with greater nuance.