PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

A new framework called Path-Aligned Decompression Distillation (PADD) aims to transfer knowledge from dense language models into more efficient mixture-of-experts (MoE) architectures, according to a paper submitted to arXiv in 2026 [1][2]. The approach addresses a central tension in large language model (LLM) development: scaling model capacity while operating within fixed computation budgets [2]. As LLMs — neural networks trained on vast text corpora for generation, summarization, and analysis — grow larger, the cost of running them at inference time becomes a binding constraint [8]. MoE models reduce that cost by activating only a subset of parameters for any given input, but training their routing policies without a pre-existing router in the teacher model has remained difficult [2]. PADD organizes the distillation process into four stages across two phases [1][2]. The initialization phase, Stage I, builds diverse functionality in the student’s experts through teacher neuron clustering and student-expert warmup [2]. The training phase, spanning Stages II through IV, integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing into a single pipeline [2]. On mathematical reasoning benchmarks, the resulting MoE student matched or surpassed its dense teacher while delivering substantial gains over strong baselines at the same inference cost [1][2]. The paper also reports stable routing behavior throughout training [2]. The work appeared on arXiv, the open-access e-print repository that has hosted preprints in physics, mathematics, and computer science since 1991 and now receives roughly 24,000 submissions per month [6].