Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

A new reinforcement-learning method called Zone of Proximal Policy Optimization (ZPPO) keeps a teacher model inside the prompt rather than the policy gradient, aiming to improve knowledge transfer to small student models, according to a paper posted to arXiv on 16 June 2026 [1]. The technique, detailed in a preprint submitted to the Computation and Language section of the open-access repository, targets a known weakness in standard knowledge distillation: when a student model is much smaller than its teacher, forcing it to imitate the teacher's output logits concentrates learning on the teacher's sharpest modes and hurts generalization on benchmarks outside the training corpus [1][2]. Reinforcement learning avoids logit imitation by training on the student's own generated responses, or rollouts, but faces a different problem. On difficult questions where every student rollout fails, those rollouts yield zero advantage and are silently discarded; injecting a teacher response into the policy gradient at that stage breaks the on-policy assumption and induces drift [1][2]. ZPPO, inspired by psychologist Lev Vygotsky's zone of proximal development, addresses this by reformulating hard questions into two new prompt types. A Binary Candidate-included Question, or BCQ, pairs one correct teacher response with one incorrect student response as anonymized candidates the student must discriminate. A Negative Candidate-included Question, or NCQ, aggregates the student's wrong rollouts into a single prompt to surface shared failure modes [1][2]. A prompt replay buffer recirculates each hard question until the student's mean rollout accuracy on it reaches one-half, or until it is evicted on a first-in-first-out basis under finite capacity, amplifying BCQ and NCQ inside the student's current zone of proximal development [1][2]. The researchers tested ZPPO on the Qwen3.5 model family at four student scales ranging from 0.8 billion to 9 billion parameters, using a 27-billion-parameter teacher [1][2]. The models were post-trained as vision-language models and evaluated on a suite of 31 benchmarks: 16 for vision-language tasks, 10 for large language model tasks, and 5 for video tasks [1][2]. ZPPO outperformed off-policy distillation, on-policy distillation, and Group Relative Policy Optimization (GRPO), with the largest gains observed at the smallest student scale [1][2]. The paper was posted on arXiv, an open-access repository that hosts electronic preprints across physics, mathematics, computer science, and related fields and that, as of late 2024, receives roughly 24,000 submissions per month [6].