The Role of Feedback Alignment in Self-Distillation

Researchers have made new findings on self-distillation and MLP residual networks, shedding light on the effectiveness of step-aligned critique and the behavior of residual streams.

Self-distillation trains a model to retain improvement when context is not present, according to a study published on arxiv.org^[1]. The method matches the model's output distribution under two settings: a student and a self-teacher. Step-aligned critique yields the largest gains in self-distillation, outperforming other methods by significant margins^[1]. Structural alignment between feedback and the solver's reasoning is a key driver of self-distillation effectiveness. Another study on arxiv.org^[2] found that the effective rank of the residual stream decreases monotonically with depth after training. The rank collapse is selective and occurs for chains with short correlation length approximately 1^[2]. The network preserves exactly the degrees of freedom relevant to the prediction task. Inter-layer kernel drift is concentrated at one or two specific transitions, and the network is near a fixed point, consistent with a discrete fixed-point plateau^[2].