Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL

A team of researchers has proposed RefGRPO, a method designed to close a persistent “reflection gap” in large language model agents by augmenting standard reinforcement learning algorithms with a free calibration bonus and a dynamic schedule, according to a paper posted to arXiv on June 12, 2026 [1]. Large language models (LLMs) are increasingly deployed as agents that interact with external environments, observing feedback such as execution results, error messages, and tool outputs [1][2]. A well-functioning agent should be able to leverage this feedback to accurately assess its own performance [1][2]. The researchers found that LLM agents tend to mis-assess their own outputs even after observing concrete environment feedback, and that standard reinforcement learning (RL) barely helps due to a credit-assignment mismatch [1][2]. To address this, the team introduced RefGRPO, which augments standard RL algorithms with two key components: a free calibration bonus computed by contrasting the agent’s own reflection with the actual outcome, requiring no additional reward model, LLM judge, or external annotation, and a dynamic schedule on its coefficient [1][2]. The paper reports that compared to standard RL baselines, RefGRPO simultaneously improves reflection calibration and task accuracy on text-to-SQL across five benchmarks [1][2]. The underconfidence rate dropped from 44.4% to 7.7%, while task accuracy rose from 75.1% to 76.5% [1][2]. The resulting calibrated reflection turns the agent into its own verifier grounded in environment feedback, which the authors say further enables better self-improvement using reflections as pseudo-rewards without outcome supervision, and more effective test-time selective prediction by committing only to rollouts flagged as correct [1][2]. The paper was posted on arXiv, an open-access repository of electronic preprints that is not peer-reviewed but is moderated before posting [6]. As of November 2024, the repository receives about 24,000 articles per month and hosts more than two million papers [6]. The work appears under the Computer Science and Artificial Intelligence categories [1].