Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

A training-free pruning method called Causal Attribution Pruning (CAP) identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks, preserving downstream accuracy better than magnitude-based criteria at equivalent sparsity levels, according to research posted to arXiv [1]. Large language models demonstrate strong multi-step reasoning but carry substantial inference costs [1]. CAP addresses this by estimating the expected performance drop when an attention head is masked during forward passes on a small calibration set of reasoning problems [1]. The resulting causal scores are converted into weight-level importance values for the corresponding projection matrices, a departure from magnitude-only or activation-based pruning criteria [1]. The approach is training-free and uses interventional measurement to capture each head's functional contribution directly [1]. Evaluations covered GSM8K, StrategyQA, and ARC-Challenge using Llama-3-8B-Instruct and Mistral-7B-Instruct at 10%, 20%, and 50% sparsity [1]. At moderate sparsity of 10-20%, CAP improved over the Wanda pruning method in most model-benchmark configurations [1]. On ARC-Challenge at 20% sparsity, CAP yielded relative accuracy gains of up to 61% over Wanda [1]. The gains were especially pronounced for Llama-3 on ARC-Challenge [1]. At 50% sparsity, the method's advantage narrowed, which the authors attribute to coarse multi-layer perceptron attribution at that level [1]. The paper appears on arXiv, an open-access repository of electronic preprints that has hosted scientific papers since August 1991 and now receives about 24,000 submissions per month [7]. arXiv papers are moderated but not peer-reviewed before posting [7]. The repository spans fields including computer science, mathematics, and physics, and many researchers self-archive papers there prior to journal publication [7]. The CAP manuscript was submitted on 27 April 2026 under the Computation and Language category [1]. LLMs are machine learning models with many parameters, trained with self-supervised learning on large text corpora for tasks such as language generation [9]. Pruning techniques aim to reduce the number of active parameters to lower inference cost, but preserving reasoning ability remains a challenge [1]. The CAP results suggest that attention-head-level causal attribution can better retain reasoning performance on downstream benchmarks than correlational pruning criteria at equivalent sparsity [1].