On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

A new study finds that on-policy self-distillation with sampled demonstrations, a technique used to boost large language model reasoning, can sharply reduce output diversity even as it maintains strong average accuracy. The work was submitted on 24 Jun 2026 by researchers including Andrei Nicolicioiu [1]. The paper, posted to arXiv, shows that while self-distilled models achieve strong pass@1 accuracy, their pass@k curves flatten, meaning generating more candidate answers fails to improve the chance of finding a correct one [1][2]. The authors trace this to compounding biases in the method’s design. During training, a single model acts as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback [2][3]. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model’s own biases [2][7]. The researchers provide a theoretical analysis of the optimal self-distillation policy. They prove that it tilts the base distribution by a pointwise conditional mutual information score between the student’s rollout and the correct rollout used as context [5][7]. In contrast, the ideal optimal on-policy reinforcement learning objective preserves probability ratios among equally correct rollouts [2][7]. Self-distillation, however, can amplify existing probability gaps and concentrate probability mass on already-dominant modes [1][7]. Experiments on a controlled graph path-finding task and science question-answering benchmarks confirmed the pattern. Self-distilled models matched or exceeded reinforcement learning on average performance but exhibited substantially lower functional and semantic diversity [1][4]. The models also failed on out-of-distribution settings that required diverse strategies [2][5]. The paper, presented at the ICML 2026 FoGen Workshop, concludes that average accuracy alone provides an incomplete picture of post-training quality and that functional and semantic diversity should be explicitly monitored when deploying such methods [4][5][7].