Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

A new method allows large language models with incompatible tokenizers to share knowledge through on-policy distillation, a technique previously confined to models within the same family, according to research submitted June 8, 2026 [1][2]. On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models [2]. However, existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series [2]. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution [2]. The new work enables the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers with a precise token-mapping algorithm [2]. Extensive experiments show that cross-tokenizer OPD is significantly more compute-efficient than baselines on various benchmarks [2]. The results unlock a broader range of teacher-student pairs for OPD, opening up new avenues for adapting and enhancing interactions between LLMs [2]. LLMs are language models with many parameters, trained with self-supervised learning on a vast amount of text [9]. The ability to distill knowledge across model families could accelerate development cycles for organizations maintaining multiple model architectures. Companies such as DeepSeek, which develops LLMs and launched its R1 model in January 2025, and Alibaba Cloud, which produces the Qwen family of models, operate in an environment where training costs and compute efficiency are critical competitive factors [8][10]. DeepSeek claims it trained its V3 model for US$6 million, far less than the US$100 million cost for OpenAI's GPT-4 in 2023, using approximately one-tenth the computing power consumed by Meta's comparable model, Llama 3.1 [8]. The paper was submitted to arXiv, a preprint server that has integrated with platforms such as Hugging Face to make machine learning research more accessible through interactive demos [6][7]. arXivLabs, a framework that allows collaborators to develop and share new arXiv features, supported the submission [1]. The research community can link papers to models, datasets, and demos on the Hugging Face Hub, and authors can claim authorship to have their work verified on their profiles [5].