Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Researchers have identified 'diversity collapse' in reinforcement learning with verifiable rewards (RLVR) for large language models, where initial improvements are followed by a decline in higher-order performance metrics.

RLVR is a key approach for enhancing the reasoning abilities of large language models, but it often suffers from diversity collapse, characterized by improving Pass@$1$ and degrading high-$k$ Pass@$k$^[1]. This phenomenon is attributed to 'overtraining,' where further updates concentrate probability mass on favored trajectories after a problem's contribution to the reference metric has saturated. Restricting updates to problems with zero observed success can improve Pass@$256$ above the base model on difficult benchmarks^[1]. Additionally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Another study found that Supervised Fine-Tuning (SFT) compresses the rollout distribution, and early Group Relative Policy Optimization (GRPO) drives the probability below a certain threshold, leading to identical rewards and no group relative signal^[2]. A two-stage diagnostic can stop failing runs early by combining pre-RL entropy triage with an early GRPO entropy monitor.