Bag of Dims: Training-Free Mechanistic Interpretability via Dimension-Level Sign Patterns

Researchers have proposed two frameworks for understanding Transformer behavior, a key component of modern AI models. The 'Bag of Dims' framework and 'Transformer Field Theory' offer new insights into how Transformers process information.

The 'Bag of Dims' framework, presented in a paper on arXiv^[1], suggests that the standard basis of transformer hidden states provides a training-free, architecture-general feature basis. Individual dimensions encode semantic content via their signs and confidence via their magnitudes. Sign patterns alone carry predictive content, achieving 72-93% top-5 next-token accuracy through the LM head^[1]. The framework discovered 175 categories via per-dimension sign consistency from 50 anchors with zero training. Static FFN weight inspection linked 20% of features to individual writer neurons with >0.70 agreement. A related paper on arXiv^[2] introduces 'Transformer Field Theory', a response-theoretic framework that views patching as localized source insertion into the Transformer field. First-order sensitivity fields predict patch effects, while Green functions describe downstream propagation. The paper poses patch selection as an adjoint inverse problem. The submission history of the 'Transformer Field Theory' paper shows it was revised on 11 Jun 2026, following its initial submission on 24 May 2026^[2].