Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short

A new training framework called Reasoning Arena aims to improve large language model reasoning by salvaging data that standard reinforcement learning with verifiable rewards discards, according to a paper posted on arXiv [1]. Reinforcement learning with verifiable rewards (RLVR) has become a leading method for enhancing reasoning in large language models, which are neural networks trained on vast text corpora for tasks such as generation and analysis [1][8]. The approach relies on outcome-based supervision, but a persistent weakness emerges when all sampled reasoning traces for a given prompt receive identical rewards. In those cases, group-relative advantage estimation provides no gradient signal, even if the traces differ markedly in quality [1]. The researchers propose Reasoning Arena, an adaptive framework that routes such non-diverse reward groups to a judge system instead of discarding them [1]. The system constructs trace tournaments, pitting reasoning traces against one another head-to-head to expose finer-grained preferences and convert reasoning quality into richer relative reward signals [1]. To keep the process computationally feasible, each new trace is evaluated against a small, dynamically updated pool of previously generated traces that serve as anchors, and a Bradley-Terry model is fitted on the resulting incomplete comparison graph. This avoids the quadratic pairwise comparisons a full tournament would require [1]. Empirical tests on competition mathematics and coding benchmarks showed Reasoning Arena outperformed the RLVR baseline by 7.6% on average [1]. The framework also accelerated training by 27% to 41% and reduced generation compute by nearly 50% by converting otherwise wasted zero-advantage samples into useful gradient updates [1]. The paper was submitted to arXiv on June 8, 2026, under the machine learning category [1]. arXiv, founded in 1991, is an open-access repository that hosts electronic preprints across physics, mathematics, computer science and other fields, and now receives roughly 24,000 submissions per month [6]. The platform also supports arXivLabs, a community collaboration framework that allows third-party developers to build experimental tools on article pages, such as bibliographic explorers and code linkers [5].