SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

Researchers have proposed new methods for on-policy distillation to improve student models in multi-turn settings, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate^[1].

On-policy distillation (OPD) trains student models on trajectories induced by their own policy, mitigating exposure bias in agent training. A new framework, SAGE-OPD, selectively applies teacher supervision based on environment feedback and teacher judgment, deciding whether to skip or intervene on student responses^[1]. SAGE-OPD also weights token-level distillation by teacher confidence and applies loss normalization to preserve the overall loss scale of standard OPD. According to arXiv preprint^[1], SAGE-OPD achieved a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Another study on arXiv^[3] found that AsyncOPD improves training throughput by 1.6x to 3.8x over strict synchronous training while reaching comparable accuracy. On-policy distillation trains a student policy using teacher signals computed on trajectories sampled by the student itself, as reported in a separate arXiv paper^[2].