BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents

A new drop-in advantage estimator called BiPACE improves policy optimization for long-horizon large language model agents by fixing a fundamental credit-assignment flaw in stepwise group-based reinforcement learning, according to research published on arXiv [1]. The method, formally titled Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation, targets what its authors describe as a state-action credit mismatch in current agentic variants of group-relative estimators [1]. These estimators typically partition steps using observation hashing, which the researchers found creates singleton groups that provide zero step-level signal. At the same time, using a single within-group mean mixes state-value estimation with action-specific credit [1]. BiPACE addresses both issues without adding a critic network, an auxiliary loss, or extra rollouts [1]. Its first component, BiGPO, clusters steps by cosine distance in the actor's own hidden-state geometry, an empirical proxy for bisimulation that substantially reduces the singleton rate left by observation hashing [1]. The second component, PACE, recenters returns within each behavioral cluster using action-conditioned peer baselines, estimating a local advantage nonparametrically [1]. On the ALFWorld benchmark using a Qwen2.5-7B model, BiPACE raised overall validation success from GiGPO's 90.8 to 97.1±0.9 over three seeds, crossing the 95% threshold on every seed, which GiGPO never did within the same training budget [1]. On a smaller Qwen2.5-1.5B model, BiPACE reached 93.5±1.2 compared to GiGPO's 86.7 [1]. Improvements over GRPO and GiGPO were also recorded on the WebShop and TextCraft benchmarks at both model scales [1]. The measured BiPACE-specific overhead is 11.3% of a single training-step wall time [1]. The code has been released on GitHub [1]. The work builds on the broader trend of training LLM agents without learned critics by reusing multiple sampled rollouts to estimate local advantages, a technique that has gained traction for long-horizon tasks where traditional value-function approximation can be unstable [2]. The paper's approach of clustering based on behavioral equivalence rather than surface-level observation identity represents a shift in how credit assignment is structured in language-agent reinforcement learning [2].