Analogies between Transformer Layers and Power Method

A new analysis draws a direct analogy between the operations inside a Transformer layer and the classic Power Method algorithm, showing that token representations are progressively tilted toward a principal eigenvector as they pass through the network [1]. The finding, posted on arXiv on 25 May 2026, focuses on the projection and layer-normalization steps within a Transformer layer, deliberately setting aside the feedforward neural network component [1]. Researchers demonstrate that these operations mirror a single step of the Power Method, a numerical algorithm used to find the dominant eigenvector of a matrix [1]. The matrix in question is formed by the product of the output and value weight matrices specific to that layer [2]. Transformers, the architecture underpinning most modern large language models, process text by stacking multiple such layers [4]. The study shows that with each successive layer, token representations are not transformed arbitrarily but are instead drawn into closer alignment with the principal eigenvector of this matrix product [2]. This alignment becomes particularly stark in a special configuration where all layers share identical weights, a phenomenon the authors confirm both through empirical observation and analytical proof [1]. The Transformer architecture has been a dominant force in natural language processing since models like BERT, introduced by Google researchers in 2018, demonstrated its effectiveness [3]. BERT, which uses an encoder-only Transformer, learns contextual representations of tokens and has become a ubiquitous baseline for NLP experiments [3]. The new research provides a mechanistic lens for understanding why such representations converge in specific ways, moving beyond treating the model as an opaque stack of layers [4]. Beyond explaining this internal dynamic, the analogy points to a practical steering mechanism. Because the layer operations mimic the Power Method, the researchers suggest it is possible to guide the Transformer's output toward an arbitrary desired direction in the token-representation space [2]. This insight could offer a more principled approach to controlling model outputs, complementing existing techniques that rely on prompting or fine-tuning [1].