Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers

Researchers have proposed Tensor Memory, a module that gives Transformer networks a fixed-size recurrent 3D memory tensor, aiming to overcome the architecture’s difficulty with long video sequences and occlusion-heavy reasoning [1][2]. Standard Transformers process images and video by flattening spatial and temporal dimensions into long token sequences. While attention mechanisms and key-value caches retain past features, the memory footprint expands with sequence length, and the model lacks an explicit, persistent spatial state [1][2]. This design complicates tasks that require tracking objects over extended periods or reasoning about occluded scenes [2]. The new module, described in a paper submitted to arXiv on 26 May 2026, introduces a constant-size memory tensor that functions as a recurrent 3D voxel grid [1][2]. Tokens write into the grid through a differentiable soft write operation, depositing content as a Gaussian-weighted volume around a predicted continuous 3D location. The memory is then updated using a local interaction operator and gated recurrent dynamics. Tokens read back context via continuous sampling with gated residual fusion [2]. Because the memory tensor maintains a fixed size, Tensor Memory decouples state capacity from input length while preserving a spatial inductive bias [1][2]. The approach draws on principles from recurrent neural networks, which maintain a hidden state updated at each time step to capture temporal dependencies [3]. Traditional RNNs struggled with long-range dependencies due to the vanishing gradient problem, a limitation later addressed by long short-term memory (LSTM) architectures introduced in 1997 and by gated recurrent units (GRUs) [3]. Transformers subsequently became dominant for many sequence-processing tasks because their self-attention mechanisms handle long-range dependencies and offer greater parallelizability [3][4]. The Transformer architecture, which gained prominence after 2017, helped fuel a broader AI boom in the 2020s alongside advances in generative AI [4]. Tensor Memory is designed to integrate with standard Transformer training pipelines and can be attached to or removed from existing blocks without other architectural changes [1][2]. The authors evaluated the module on language, image, and video benchmarks, as well as on a controlled toy diagnostic suite built to isolate conditions where persistent state proves beneficial [2]. The work arrives as major AI laboratories continue to explore architectures that blend attention with external memory. In 2014, Google DeepMind introduced neural Turing machines, which gave neural networks access to external memory resembling a conventional Turing machine [5]. DeepMind, now a subsidiary of Alphabet, has since developed models ranging from AlphaGo to the Gemini family of large language models [5].