From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models

A new study finds that common observational metrics used to assess the importance of individual experts in Mixture-of-Experts language models do not predict their actual causal role, challenging a widespread practice in model interpretability and pruning [1]. Researchers from the Pearl lab conducted a token-level interventional audit across three high-redundancy MoE architectures: OLMoE-1B-7B-0924, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite [1]. The team tested whether routing statistics such as utilization rates, activation norms, and routing weight distributions could identify which experts could be removed without degrading model function [1]. After applying multiple-comparison correction, no observational metric reached statistical significance in any model. Effect sizes remained below Cohen's d of 0.17 across all 60 metric-layer combinations examined [1]. The work draws on Judea Pearl's causal hierarchy, arguing that interpretability researchers routinely treat associational evidence—what Pearl calls rung-1—as if it supported interventional, or rung-2, conclusions [2]. The paper frames MoE pruning as a concrete test case for this inferential leap, which the authors say is rarely validated [3]. To rule out the possibility that the null result stemmed from insufficient statistical power, the researchers applied a per-token routing weight control using identical methodology and sample size [4]. This control recovered a single Bonferroni-significant signal at OLMoE's final MoE layer, with a Cohen's d of +0.231 and a p-value of 0.0013 [1]. At that layer, ablating an expert shifted the residual stream by roughly 40 times the magnitude observed at early layers [4]. The effect was unique to OLMoE; neither Qwen1.5-MoE-A2.7B nor DeepSeek-V2-Lite exhibited comparable concentration [4]. The authors treat the OLMoE late-layer signal as a within-model regularity and caution that three architectures are insufficient to attribute it to any single cause [4]. The findings carry implications for MoE pruning research, where expert-dropping techniques have proliferated. A separate study on MoE lottery subnetworks noted that iterative pruning with task-agnostic fine-tuning can stabilize model performance after expert removal, but also observed that instruction-following capabilities are disproportionately harmed during the process [5]. The Pearl lab's audit suggests a different mechanism: existing pruning methods succeed not because they identify genuinely dispensable experts, but because early-layer redundancy makes most selection criteria interchangeable [1]. The paper provides what the authors describe as an explicit counterexample to the common step from population-level observational summaries to token-level interventional claims about expert importance [2]. It also illustrates how interventional audits can calibrate the evidential standards required for interpretability claims in large language models [3].