Not All Transitions Matter: Evidence from PPO

A new study finds that randomly discarding a quarter of the experience samples during training can make reinforcement learning agents more stable without hurting their final performance, challenging the assumption that more data is always better [1]. The work, posted to arXiv on 22 May 2026, targets a subtle flaw in on-policy algorithms like Proximal Policy Optimization (PPO). When an agent interacts with an environment, each new state is a direct consequence of the previous one, creating a causal chain. This means consecutive transitions are not independent; they carry overlapping information that can cause the network to receive a repetitive gradient signal, leading to instability that reward curves often mask [1][2]. The researchers propose a minimal intervention: randomly drop a fixed fraction of transitions from the collected rollout before the network update, while preserving the reward signal structure. The method requires no new components and works with any standard PPO implementation [1][2]. Tests across five benchmark environments of increasing complexity—CartPole-v1, Acrobot-v1, LunarLander-v2, HalfCheetah-v5, and Hopper-v5—showed the approach matched the reward of vanilla PPO. Crucially, it produced more consistent training dynamics when measured by KL divergence, policy entropy, and value estimates [1][2]. The authors identified 25% as the effective drop rate, describing it as enough to disrupt the redundancy in the batch without thinning the data to the point of harming learning [1][2]. The paper argues that the repetitive gradient structure, where the same directions are reinforced repeatedly, is a root cause of quiet training instability, and that simply removing some of that redundancy is sufficient to break the cycle [2]. The technique is presented as a single additional sampling step that can be inserted into existing PPO pipelines. By demonstrating that not all transitions contribute equally to stable learning, the findings suggest a path toward more reliable training dynamics without architectural changes or hyperparameter tuning beyond the drop fraction [1][2].