EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

A new framework called EfficientRollout reduces the latency of reinforcement learning rollouts for large language models by up to 19.6% without degrading final model quality, according to research submitted on 17 Jun 2026 [1]. The work targets a persistent bottleneck in post-training LLMs with reinforcement learning (RL), a paradigm that has enabled strong reasoning and agentic capabilities [2]. During RL, rollout generation—where the model produces responses through autoregressive sampling—dominates latency because a small number of long-tailed generations often determine completion time [2]. Speculative decoding (SD) is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution [2]. However, its practical speedups do not directly carry over to RL rollouts for two reasons: the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution, and active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute [2]. EfficientRollout addresses this gap through a system-aware self-speculative decoding framework. It induces a quantized drafter directly from the target model, a technique known as self-speculative decoding, which keeps the drafter coupled to the evolving policy without requiring separate drafter pretraining or online adaptation [2]. The framework further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality [2]. Over an accelerated autoregressive rollout baseline, EfficientRollout reduces rollout latency by up to 19.6% and end-to-end latency by up to 12.7% [1][2]. The researchers report that final model quality is preserved [1]. The submission, posted to arXiv on 17 Jun 2026, appears in the Machine Learning section of the repository [1]. The paper's abstract notes that RL has become a representative post-training paradigm for LLMs, and that rollout generation remains a dominant latency bottleneck [2]. The EfficientRollout framework is designed to accelerate these rollouts by combining a drafter that remains effective under long, high-temperature generations from an evolving policy with system-aware use of speculative decoding that avoids compute-bound regimes [2].