Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution

A new sparse autoencoder architecture that operates on averaged conversation turns rather than individual tokens could make interpreting large language models practical at long context lengths, according to research submitted on 26 June 2026 [1]. Sparse autoencoders, or SAEs, have become a standard tool for extracting interpretable features from language model activations. But standard SAEs process one token at a time, so the number of active features grows linearly with context length, making analysis of long transcripts unwieldy [1][2]. The proposed turn-averaged SAE instead learns to reconstruct the mean hidden-state activation across an entire Human or Assistant turn, producing a fixed number of features regardless of how many tokens the turn contains [2][3]. The training objective remains standard SAE reconstruction loss; the only change is the input representation [2][3]. When evaluated by a language model judge, turn-averaged features described a turn's high-level characteristics more completely than per-token features [1][2]. The architecture also simplifies downstream interpretability tasks. For attribution graphs, turn-averaged features yield nodes representing features per turn per layer rather than per token per layer. A 10-turn conversation with 250 tokens across four layers produces roughly 5,000 turn-averaged candidate nodes, compared with approximately 128,000 per-token nodes [2][3]. The mean hidden state retains the same dimensionality regardless of span length, so turn-averaged SAEs should generalize to contexts far longer than those seen during training [2][3]. The researchers tested this by encoding documents from LongBench v2, which range from 24,000 to 32,000 tokens. Each document was treated as a single span, roughly 150 times longer than the average training example and drawn from a different corpus. The turn-averaged SAE identified high-level concepts within the span more reliably than the per-token SAE, suggesting the learned characteristics generalize across span lengths even at scales far beyond the training data [2][3]. Related work has explored temporal structure in SAE training. Temporal Sparse Autoencoders, introduced in a 2025 preprint, incorporate a contrastive loss that encourages consistent activations of high-level features over adjacent tokens, disentangling semantic from syntactic features without explicit semantic supervision [4]. The turn-averaged approach takes a different path, collapsing temporal information into a single mean representation per turn rather than modeling token-to-token consistency [1][2].