Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

A new analytical framework called the Transformer Geometry Observatory (TGO) has been introduced to systematically probe the internal representational geometry of Vision Transformers, an area its creators describe as relatively underexplored compared to the models' widespread application success [1]. The framework's first installment, designated TGO-I, concentrates on the spectral geometry of Vision Transformer representations [1]. Researchers deployed a ViT-Small/16 architecture trained on the ImageNet-100 dataset to track a suite of metrics throughout the training process, including Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, and Spectral Anisotropy [2]. The analysis also examined covariance structure, eigenspectra, and singular value spectra across all layers and training epochs [3]. The results documented a consistent increase in dimensional utilization as training progressed, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra [1]. Covariance matrices became increasingly diagonal, indicating progressively weaker feature correlations [4]. This pattern runs counter to the common intuition that training should concentrate information into a small number of dominant directions; instead, the team observed a progressive redistribution of variance across representational dimensions [2]. The effect was most pronounced in the final CLS token representation, which consistently exhibited the highest effective dimensionality and lowest anisotropy within the network [1]. In contrast, the Patch Embedding and Positional Embedding layers remained comparatively stable throughout training, suggesting that the majority of geometric evolution occurs within the Transformer processing blocks and the final global representation [5]. The research was posted on the arXiv preprint server, an open-access repository that hosts scientific papers in fields including computer science and has grown to a submission rate of about 24,000 articles per month as of late 2024 [9]. The TGO framework is presented as a surgical set of experiments and analysis pipelines built to analyze the representational geometry and dynamics of Vision Transformers, with TGO-I serving as the inaugural observatory within the broader project [3].