Emergent Analogical Reasoning in Transformers

New research provides a mechanistic account of how Transformer neural networks perform analogical reasoning, a capability long considered a hallmark of human intelligence. The study, posted to arXiv, identifies two distinct computational stages underlying the process. The paper, authored by Gouki Minegishi and revised on 23 May 2026, formalizes analogical reasoning as the inference of correspondences between entities across different categories, drawing on the mathematical concept of functors from category theory [1, 2]. The work aims to move analogy from an abstract cognitive notion to a concrete, mechanistically grounded phenomenon in modern neural networks [2]. Artificial intelligence research has historically pursued capabilities including reasoning, learning, and perception, often using artificial neural networks [3]. The Transformer architecture, introduced in 2017, accelerated progress in the field and underpins the large language models that became widely available in the 2020s [3]. Through mechanistic analysis using controlled synthetic tasks, the researchers found that analogical reasoning in Transformers decomposes into two key components. The first is a geometric alignment of relational structure within the model's embedding space. The second is the application of a functor inside the Transformer, which enables the model to transfer relational structure from one category to another, thereby realizing an analogy [1, 2]. The study also reports that the emergence of this reasoning capability is highly sensitive to specific data characteristics, optimization choices, and the overall scale of the model [1, 2]. These findings were not limited to purpose-built experimental models; the researchers quantified the same mechanistic trends in pretrained large language models [1, 2]. The paper was originally submitted on 2 February 2026 and underwent several revisions, with the latest version posted on 23 May 2026 [1].