Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

New research on Meta’s Llama-3.2 models shows that pruning the width of their feed-forward layers does not degrade all capabilities uniformly. Instead, the study identifies a “dichotomy” where factual knowledge declines while instruction-following improves at a specific compression ratio [1][2]. The paper, posted on the arXiv preprint server and last revised on 12 June 2026, examines structured width pruning of GLU-MLP layers using a Peak-to-Peak Magnitude (PPM) criterion [1][2]. Researchers evaluated seven different expansion ratio configurations across two model sizes, Llama-3.2-1B and Llama-3.2-3B [2]. The work was submitted by Pere Martra on 27 December 2025, with subsequent revisions on 6 May and 12 June 2026 [1]. Performance on benchmarks that test parametric knowledge, such as MMLU and GSM8K, fell predictably as the expansion ratio was reduced [2]. However, instruction-following scores, measured by IFEval, rose at what the authors call the 2.4x equilibrium ratio. For the 1-billion-parameter model, IFEval improved by 4.8 points, or 46 percent, while the 3-billion-parameter version gained 3.7 points, or 39 percent [2]. Multi-step reasoning, tracked by the MUSR metric, remained robust across the tested configurations [2]. The findings challenge a common view in model compression research that pruning causes uniform performance loss. The authors argue that the expansion ratio acts as a critical architectural parameter that selectively reshapes a model’s task profile rather than simply serving as a compression metric [2]. The paper appears on arXiv, an open-access repository that hosts preprints in fields such as computer science and physics and has grown to receive roughly 24,000 submissions per month as of late 2024 [6]. The study is listed under the Computation and Language category and is accompanied by bibliographic and code-discovery tools available through the arXivLabs framework, a community-collaboration initiative that provides third-party features on article pages [1][4][5].