Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models

A new method called Sparrow uses dynamic sparse attention to accelerate the training of large language models under reinforcement learning with verifiable rewards, or RLVR, according to research posted to arXiv on June 7, 2026 [1][2]. The technique achieves rollout speedups of 2.2x, 2.4x, and 2.0x for three model sizes while maintaining training stability [1][2]. RLVR is a powerful training paradigm but induces extremely long chain-of-thought reasoning, making the process computationally expensive [1][2]. The dominant per-step cost comes from generating these long-context rollouts, and sparse attention has been proposed as a way to accelerate the dense rollout generation [2]. However, the authors note a fundamental tension: overly aggressive sparsity causes training collapse, while overly lenient sparsity provides insufficient speedup [1][2]. The Sparrow method addresses this by studying what the researchers call the sparse-to-dense actor-policy mismatch [2]. They observed that collapse is not driven by uniform degradation across all tokens; most sparse tokens align perfectly with their dense counterparts even under aggressive sparsity [2]. This led to the hypothesis that training remains stable if the lower tail of the per-token mismatch stays above a critical threshold throughout the trajectory [2]. The team introduced a dynamic sparsity schedule that keeps this tail statistic constant during generation [1][2]. They validated the approach across the Qwen3 thinking-family models, finding that maintaining the tail mismatch near a consistent threshold generally enabled stable training [2]. A cost model was then used to find the sparsity schedule that maximizes speedup under that mismatch threshold, yielding the reported gains for the 1.7-billion, 4-billion, and 8-billion parameter versions [2]. The researchers also showed empirically that the thresholds generalize to a larger 14-billion-parameter model and to a separate reinforcement-learning domain, coding [2]. The analysis further motivated a technique called DistillSparse, a lightweight LoRA-based distillation on sparse rollouts that allows more aggressive sparsity to reach the same mismatch threshold, yielding even higher speedup [2]. The paper was submitted to arXiv, an open-access repository of electronic preprints that, as of late 2024, receives about 24,000 articles per month and is not peer-reviewed [6].