TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts

A new structured pruning framework called TENP aims to reduce the deployment footprint of Mixture-of-Experts large language models without the performance penalties of prior compression methods, according to a paper posted to the arXiv preprint server [1]. The framework, formally named Trapezoidal Expert Neuron Pruning, targets the large static parameter footprint that constrains Mixture-of-Experts (MoE) models despite their efficient sparse activation during inference [1][2]. Existing approaches either remove entire experts, which disrupts routing topology, or rely on unstructured weight pruning that offers limited practical efficiency [2]. TENP instead identifies and retains important experts using a small number of samples, then applies expert neuron pruning to less important experts while reserving parameters in a trapezoidal pattern from shallow to deep layers [1][2]. The importance of an expert is evaluated by jointly considering the magnitude of its output and its ability to change the direction of the input vector [2]. The researchers tested TENP on the Qwen and DeepSeek model families [1][2]. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model experienced only a 1-point drop in accuracy compared to the full-parameter baseline [1][2]. On code generation tasks, the pruned model outperformed the full-parameter version by 10% [1][2]. The paper was submitted to arXiv on 3 June 2026 under the machine learning category [1]. arXiv, which began operating in August 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed; it surpassed two million articles by the end of 2021 and now receives roughly 24,000 submissions per month [6]. The platform also hosts arXivLabs, a framework launched in 2020 that allows community collaborators to build experimental tools on top of the repository while adhering to values of openness and user data privacy [5]. Large language models, the class of neural networks that MoE architectures fall under, are typically based on the transformer architecture and are pre-trained on vast text corpora to perform tasks such as generation, summarization, and translation [8].