Explaining Attention with Program Synthesis

A team of researchers has proposed a method to explain the inner workings of transformer language models by replacing their attention heads with human-readable Python programs, a step toward making deep learning systems more transparent. The approach, detailed in a paper submitted on 17 June 2026, targets the multi-head attention mechanism that is central to transformer architectures [1][4]. Transformers, first proposed in the 2017 paper "Attention Is All You Need," convert text into numerical tokens and use parallel attention heads to amplify key signals and diminish less important ones [4]. The new method seeks to reverse-engineer these attention heads by generating executable code that mimics their behavior [1][2]. The pipeline works in three stages. First, researchers compute attention matrices for a given head using randomly selected training examples. Next, they prompt a pre-trained language model with a summary of those matrices and instruct it to generate a set of Python programs that can reproduce the attention patterns from input text alone. Finally, the programs are re-ranked based on how well they predict behavior on held-out inputs [1][2]. The researchers tested the technique on GPT-2, TinyLlama-1.1B, and Llama-3B [1][2]. Fewer than 1,000 generated programs were sufficient to reproduce attention patterns, achieving an average Intersection-over-Union similarity above 75% on the TinyStories dataset [1][2]. When the best-fit programs were used to replace neural attention heads, the impact on model performance was limited. Replacing 25% of attention heads with programmatic surrogates across the three models resulted in only a 16% average increase in perplexity, while performance on downstream question-answering benchmarks was maintained [1][2]. Deep learning, a subfield of machine learning based on artificial neural networks with multiple processing layers, has driven advances in natural language processing, computer vision, and generative artificial intelligence [6]. The push for interpretability has grown alongside the deployment of these models in high-stakes settings. The new work contributes a scalable pipeline for producing human-readable, executable descriptions of neural components, advancing what the authors call a path toward symbolic transparency [1][2].