Large language models reorganize representational geometry during in-context learning

Large language models reorganize their internal representational geometry during in-context learning, a process that increases the separability of task-relevant information, according to a paper submitted to arXiv on 16 May 2026 [1]. The study examines how large language models (LLMs) adapt to novel tasks from in-context examples without parameter updates, a capability known as in-context learning (ICL) [1]. Prior work demonstrated that ICL can implement specific algorithms and identified key circuits supporting the behavior, but the role of the high-dimensional representation space remained unclear [2]. The authors hypothesized that ICL depends on the successful online untangling of task-relevant representations, a perspective borrowed from neuroscience where classification is viewed as the untangling of neural representations [2]. To test this, they studied how LLMs classify in-context examples whose labels are defined by the model's own internal representations with known structure [1]. The results showed that ICL performance correlates systematically with the representational structure of the underlying classification task [2]. Successful ICL was accompanied by geometric reorganization that increased online separability, meaning the model reshapes its internal representation space to better distinguish between categories as it processes examples [1]. The paper further finds that LLM behavior is well described by a prototype-like algorithm that integrates evidence while reshaping representations to support classification [2]. These findings offer a geometric account of ICL in pretrained LLMs and establish representational geometry as a mechanistic constraint on the process [1]. The work quantifies the gap between what pretrained representations afford and what in-context learning can exploit, suggesting that the initial geometry of the model's representation space sets limits on how effectively it can adapt to new tasks on the fly [2]. The paper was submitted on 16 May 2026 to arXiv, a preprint server for research in computer science and related fields [1].