The Reservoir Attention Network: Cross-Pass State in Pretrained Transformers via Content-Addressable Reservoir Injection

A new study introduces the Reservoir Attention Network, an architecture that injects a fixed, randomly initialized reservoir into a pretrained transformer’s mid-layer attention to carry state across forward passes [1][2]. The work, posted to arXiv on 14 June 2026, tests the concept on models from GPT-2 to Qwen2.5 using a single consumer GPU [1][2]. The Reservoir Attention Network (RAN) is designed to isolate whether untrained recurrent dynamics alone can carry usable cross-pass state [1][2]. The reservoir is left untrained by design, a deliberate choice that leaves trained recurrence as a complementary but more computationally expensive direction [1][2]. The experiments span GPT-2 at 124M and 355M parameters, and Qwen2.5 at 0.5B and 1.5B parameters [1][2]. The authors describe the tasks as minimal probes chosen to isolate individual mechanisms, and they frame the broader vision of an always-alive agent as compute-limited future work rather than a claim of the current paper [1][2]. Large language models such as GPT-2 and Qwen2.5 are typically built on the transformer architecture, which is more parallelizable than earlier recurrent neural network models [3]. Generative pre-trained transformers are pre-trained to predict the next word and are often fine-tuned to follow instructions [3]. The RAN study explores whether injecting a fixed recurrent component into such models can extend their ability to maintain state across forward passes without additional training of the recurrent element [1][2]. The paper appeared on arXiv, an open-access repository of electronic preprints that is moderated but not peer-reviewed [7]. As of November 2024, the repository receives about 24,000 new articles per month and has hosted more than two million articles since its founding in 1991 [7]. The RAN preprint is accompanied by arXivLabs integrations, a framework that allows community collaborators to develop and share experimental tools directly on the arXiv abstract page [5][6]. These tools include bibliographic explorers, code finders, and recommender systems that help readers navigate related research [6]. arXivLabs projects operate under guidelines that require partners to share arXiv’s values of openness, community, excellence, and user data privacy [5]. The RAN study does not include trained recurrence; the fixed reservoir is intended to test the lower bound of what recurrent dynamics can contribute without learned parameters [1][2]. The authors note that trained recurrence remains a complementary direction, one that would require greater computational resources [1][2]. The work is presented as a feasibility and dynamics study, with the always-alive agent concept treated as a long-term goal that exceeds the scope of the current experiments [1][2].