When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training

A new algorithm called Evidence-Calibrated Policy Optimization (ECPO) aims to make reinforcement learning more reliable for long-horizon tasks performed by large language model agents, according to a paper submitted June 4 [1]. The method addresses a known weakness in group-based credit assignment, where intermediate decisions in a task sequence receive step-level rewards. Researchers found that under limited rollouts, rare but successful actions can receive outsized advantages, leading to training instability and what they term divergent anchor bias [1]. ECPO is designed as a critic-free algorithm that calibrates this step-level credit before each policy update [1]. It does so through two mechanisms. Evidence-Calibrated Action Advantage groups rollouts by canonical actions and shrinks estimates that rely on low sample counts. Variance-Gated Credit Weighting then suppresses anchor states where within-action noise dominates the signal [1]. The combined approach adds only 0.1% additional advantage-computation overhead compared to the GiGPO baseline [1]. In experiments using the Qwen2.5-1.5B model, ECPO improved success rates on the ALFWorld and WebShop benchmarks by 5.2 and 7.3 points, respectively, over GiGPO [1]. Qwen is a family of models developed by Alibaba Cloud, with several versions distributed under open-source licenses such as Apache 2.0 [4]. The 1.5-billion-parameter variant used in the tests falls into the category of small language models, which typically range from a few thousand to a few hundred million parameters and can be trained on more modest hardware than their trillion-parameter counterparts [6]. Long-horizon agent tasks present a challenge because rewards are often sparse and delayed, making it difficult to determine which earlier actions contributed to a final outcome [1]. Group-based methods like GiGPO attempted to solve this by constructing advantages at repeated anchor states, but the ECPO authors argue that statistical unreliability in those dense credit signals can cause late-stage oscillation during training [1]. By calibrating credit with evidence-based shrinkage and variance gating, ECPO seeks to stabilize the learning process without introducing a separate critic network [1].