MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

A team of researchers has proposed Mixture of Slimmable Experts (MoSE), a new architecture for large language models that allows for more granular control over computational resources by executing individual experts at variable widths, according to a paper posted on arXiv [1]. Standard Mixture-of-Experts (MoE) models improve efficiency by sparsely activating a subset of experts for each input, but once an expert is chosen, it runs at full capacity. This creates large, discrete jumps in the trade-off between accuracy and computation [1]. The MoSE architecture, introduced by Nurbek Tastan and colleagues, addresses this by giving each expert a nested, slimmable structure that can be executed at different widths [2]. This decouples expert selection from expert capacity: the router still decides which experts are active, but a new mechanism controls how much of each expert is utilized [4]. The result is that a single pretrained MoSE model can support a continuous spectrum of accuracy-compute trade-offs at inference time, without retraining or altering the expert parameters [5]. The researchers detail a training recipe that combines multi-width training with standard MoE objectives, which they describe as simple and stable [2]. During inference, they explore methods for determining the optimal width at runtime. One approach involves a lightweight test-time training procedure that learns to map router confidence scores to expert widths under a fixed compute budget, keeping all model parameters frozen [4]. Experiments were conducted on GPT-style models using various routing regimes and evaluated on zero-shot downstream reasoning benchmarks. The team also tested the architecture through continual pre-training adaptation of the DeepSeek model [2]. The findings show that MoSE matches or improves upon the performance of a standard MoE when operating at full width. More significantly, the architecture consistently shifts the compute-quality frontier, achieving comparable performance with fewer floating-point operations [3]. The paper was initially submitted on 5 February 2026, with a revised version posted on 16 June 2026 [1]. The code for the project has been made available on GitHub [2].