Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

A new lightweight augmentation for autoregressive transformers, called the Latent Recurrent Transformer (LRT), improves language-modeling performance and in-context learning while adding as little as 0.3% to the model's parameter count, according to a paper submitted to arXiv on 26 May 2026 [1]. The LRT architecture introduces a cross-layer recurrent latent pathway that reuses a high-level hidden state from the previous token as memory for the next token. Because this source state is already computed during standard decoding, the method does not require inserting pause tokens or extra depth loops, and it preserves the standard attention mechanism and KV-cache interface [1]. This design stands in contrast to earlier recurrent neural networks (RNNs), which were foundational for modeling sequential data such as speech and time-series information but were largely superseded by transformer architectures that introduced attention mechanisms to handle long-range dependencies [3]. To train the recurrence at scale, the researchers developed a technique called interleaved parallel training. A single full-sequence initialization forward pass builds a shared buffer, after which disjoint position subsets are refined in parallel and written back. This process ensures all tokens receive recurrent-memory-aware supervision at roughly 2 times the baseline compute cost [1]. The approach avoids the computational bottleneck of sequentially unrolling the transformer, a challenge that has historically made training deep neural networks a compute-intensive process accelerated by graphics processing units and large datasets [3]. Across nanochat-style backbones and a wide range of tokens-per-parameter budgets, LRT improved both language-modeling loss and in-context learning under matched effective compute [1]. The minimal parameter overhead—as little as 0.3%—is notable given that deep learning architectures typically range from three to several hundred or thousands of layers, with each layer performing a transformation on its inputs [4]. The LRT pathway operates by routing information across layers without expanding the model's structural footprint, a concept loosely analogous to how autoencoders learn efficient codings of data by transforming inputs into lower-dimensional representations for subsequent use [5]. The paper was posted on arXiv under the Computer Science and Machine Learning category and is associated with arXivLabs, a framework for experimental projects with community collaborators [1].