Learning State-Tracking from Code Using Linear RNNs

Linear recurrent neural networks can track state in code-based tasks where Transformer models continue to fall short, according to a revised preprint posted to arXiv on 25 June 2026. The work reframes a classic sequence-modeling benchmark to better reflect how language models are trained. The study, led by Julien Siems, converts permutation-composition problems — a standard testbed for sequence models — into code traces that interleave state reveals through print statements and variable transformations [1][2]. This representation aligns the task with the next-token prediction objective used to train large language models, rather than the sequence-to-sequence format of earlier benchmarks [2]. Transformers, which rely on multi-head attention mechanisms to contextualize tokens within a fixed context window, have dominated language modeling since the 2017 “Attention Is All You Need” paper [3]. Their parallel architecture reduced training time compared to earlier recurrent designs such as long short-term memory networks [3][6]. Yet the new experiments show that even state-of-the-art Transformer models fail to track state reliably in the code-based setup, while linear RNNs succeed [1][2]. The paper also identifies a harder regime. When actions are not always fully observable, the problem becomes one of tracking a probabilistic finite-state automaton with deterministic state reveals [2]. In that setting, linear RNNs can underperform their non-linear counterparts [1][2]. Recurrent neural networks, which process sequential data by passing signals through loops of artificial neurons, have long been used for speech and time-series modeling [6]. Non-linear activation functions give them representational capacity that purely linear recurrences lack, a distinction that matters when observations are incomplete [2][6]. The preprint was first submitted on 16 February 2026 and revised twice, with the latest version posted on 25 June 2026 [1]. The findings add to a growing body of work probing the architectural limits of Transformers, which underpin systems from chatbots to code-generation tools [3][5]. Since the 2020s, generative AI built on Transformer backbones has been widely deployed, but the new results suggest that alternative architectures may be necessary for tasks requiring robust internal state maintenance [2][5].