VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoring

A new training method for sparse autoencoders, called VASAE, directly links the internal features of Transformer language models to specific tokens in the model's vocabulary, according to a preprint posted to arXiv [1]. The method, short for Vocabulary-Aligned Sparse Autoencoder, trains SAE features under a constraint called vocabulary-aligned anchoring. This process assigns each learned feature an intrinsic token name, defined as the token string whose embedding is closest to that feature's direction in the model's representation space [1][2]. The work addresses a known limitation in mechanistic interpretability, where researchers typically must label the features discovered by an SAE after training is complete [2]. In experiments on GPT-2-small, the team applied a 0.8 cutoff on a nearest-token alignment score and found that approximately 90% of features in layers 0 through 10 aligned with a vocabulary token [1][2]. The approach was also tested on the larger Llama-3.1-8B model. A representative shallow-layer dictionary achieved 92.8% feature alignment, while a middle-layer dictionary also showed strong alignment. However, a dictionary from the model's final layer exhibited limited alignment, suggesting the method's effectiveness varies with network depth [1][2]. The researchers report that VASAE achieves this alignment without reducing reconstruction quality compared to a standard sparse autoencoder [1][2]. In a further analysis, the authors subtracted the sentence-level mean sparse code and conducted case studies. They observed that many of the remaining intrinsic token names were relevant to nearby input tokens, indicating that the assigned names carry contextual meaning [2]. The preprint, titled "VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoring," was submitted to arXiv on June 26, 2026 [1]. The authors conclude that vocabulary-aligned anchoring can connect learned features to intrinsic token names during training, offering a complement to post-hoc interpretation techniques for understanding the internal computations of large language models [2].