On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

Researchers have detailed a new distillation method, Guided On-Policy Distillation (Guided-OPD), designed to improve the performance of smaller language models acting as multi-turn agents by preventing early errors from derailing the learning process [1]. The algorithm addresses a core weakness in standard On-Policy Distillation (OPD) for multi-turn agents. In conventional OPD, a small student model's errors can compound across sequential turns, pushing the interaction into states the larger teacher model was not trained to handle. This means the teacher's supervision becomes least reliable when the student needs it most [2]. Guided-OPD counters this by mixing teacher-generated and student-generated turns within a single task rollout. A curriculum schedules the teacher's intervention probability, starting with strong guidance to keep early trajectories close to the teacher's familiar distribution and then gradually decaying the intervention to zero, recovering a purely on-policy regime for inference [2]. The approach was tested across three interactive environments: ALFWorld, ScienceWorld, and WebShop. Researchers distilled Qwen3 student models from a Qwen3-30B-A3B teacher [2]. On average, Guided-OPD improved the Score metric by 21.1% and the Success Rate by 25.5% over vanilla OPD, with the gains described as larger on the smallest student models [2]. The work was submitted as a preprint on June 14, 2026 [1]. Large language models, which underpin such agents, are trained on vast text corpora using self-supervised learning and can contain billions of parameters [11]. Their operational cost makes distillation into smaller, more efficient models a key area of research. The preprint appeared on arXiv, an open-access repository that hosts over two million scholarly articles and receives roughly 24,000 new submissions per month, serving as a primary distribution channel for computer science and machine learning research [9]. The platform also supports community-built tools through its arXivLabs framework, which allows third-party developers to create features such as citation explorers and code finders that integrate directly with paper abstract pages [7][8].