Demystifying Variance in Circuit Discovery of LLMs

Researchers have proposed new methods for circuit discovery in large language models (LLMs), a key technique in mechanistic interpretability. The current state-of-the-art method, EAP-IG, has been improved upon by CEAP, which offers a theoretical guarantee.

Circuit discovery is crucial for identifying model components vital for specific tasks. EAP-IG, the current state-of-the-art, suffers from variability issues, including resampling and rephrasing variance^[1]. CEAP, a new method submitted on June 15, 2026, lessens resampling variance and improves upon EAP-IG with a theoretical guarantee^[1]. Existing circuit discovery approaches rely on iterative edge pruning, which is computationally expensive and limited to coarse-grained units. A proposed method introduces learnable masks across multiple granularity levels, from blocks to individual neurons, and has a 5-10x lower memory footprint compared to prior methods^[2]. Researchers argue that LLMs may be inherently hard to steer due to rephrasing variance, as different templates activate different circuits. Sparsity fails to solve this problem, while sample-wise variance is considered largely benign^[1].