Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation

A new study of Mixture-of-Experts language models finds that multilingual training diffuses expert routing in early layers while concentrating language-specific behavior in the final layers, and proposes a lightweight adaptation method that updates less than 2% of model parameters. The paper, submitted on 28 May 2026 to arXiv, examines how expert usage shifts when an English-centric Mixture-of-Experts model undergoes continual pre-training on a multilingual corpus [1]. Mixture-of-Experts architectures are a widely adopted technique for scaling neural language models by activating only a subset of specialized sub-networks, or “experts,” for each input [2][3]. The researchers report that early and middle layers develop diffused, language-agnostic routing patterns, while language specialization emerges primarily in the final layers [1]. Token-level vocabulary overlap between languages was identified as a significant factor influencing routing decisions [1]. Building on these observations, the team designed a parameter-efficient adaptation strategy that updates only language-specific and shared experts within the final MoE layers [1]. The approach was evaluated on the MultiBLiMP and Belebele benchmarks, where it achieved competitive performance compared to fine-tuning complete final layers while modifying less than 2% of the model’s total parameters [1]. The findings offer a practical pathway for extending large language models to low-resource languages without the computational cost of full retraining. Neural networks, the broader family of models to which MoE architectures belong, consist of layers of interconnected artificial neurons that transform input signals through weighted connections learned during training [3]. Modern large language models are predominantly built on transformer architectures, which use attention mechanisms to capture long-range dependencies in text [3]. The study’s focus on routing dynamics adds to a growing body of work examining how internal model components specialize during multilingual learning. The research contributes to ongoing efforts to make language technology more accessible across linguistic communities. By identifying precisely where language specialization occurs and demonstrating that targeted updates to a small fraction of parameters can yield strong results, the work reduces the barrier to adapting powerful models for languages with limited digital resources [1]. The code associated with the paper has been made publicly available on GitHub [1].