A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation

A new study proposes a training recipe that combines on-policy optimization with distillation to improve long-context reasoning in large language models, addressing limitations in existing post-training methods. The work, revised on 16 Jun 2026, introduces a method called Distilled Group Relative Policy Optimization (dGRPO) and a synthetic dataset named LongBlocks [1][3]. The research, led by Miguel Moura Ramos, identifies complementary weaknesses in three common approaches for adapting large language models (LLMs) to tasks requiring coherence over thousands of tokens [1]. Supervised fine-tuning (SFT) offers stable supervision but suffers from exposure bias, while reinforcement learning methods like Group Relative Policy Optimization (GRPO) struggle with long-horizon credit assignment and sparse rewards [1][2]. On-policy distillation (OPD) provides dense token-level guidance but does not directly optimize task rewards [1][4]. Reasoning models, a class of LLMs trained for multi-step logical tasks, are the intended beneficiaries of such long-context alignment techniques [6]. The proposed dGRPO method augments GRPO with dense guidance from a stronger teacher model via OPD [3][4]. In this setup, the student model learns from its own generated trajectories using outcome-level rewards, while the teacher provides token-level regularization in place of a standard reference policy [1][2]. The paper notes this is especially useful when process-level supervision is difficult to obtain [1][2]. The combined objective stabilizes training: experiments showed that while GRPO increased reward but remained noisy, and OPD yielded smaller gains, dGRPO achieved the smoothest training, the highest final reward, and better long-context performance while preserving short-context accuracy [4]. To support the study, the authors created LongBlocks, a synthetic multilingual dataset covering multi-hop reasoning, contextual grounding, and long-form generation [1][2]. Controlled ablations isolated the roles of cold-start initialization, teacher anchoring, and data mixing [1][2]. The recipe relies on an off-policy cold start to improve long-context optimization, using the LongBlocks dataset for both off-policy warm-up and on-policy alignment [4]. A separate 2025 investigation into OPD dynamics found that successful distillation requires the student and teacher to share compatible thinking patterns and that the teacher must offer genuinely new capabilities [5]. That work proposed off-policy cold start and teacher-aligned prompt selection as strategies to recover failing OPD configurations [5]. The dGRPO recipe incorporates these insights, using an SFT warm-start phase on teacher-generated rollouts to bridge thinking-pattern gaps before on-policy training begins [4][5].