RotRNN: Modelling Long Sequences with Rotations

A new linear recurrent model called RotRNN uses rotation matrices to simplify long-sequence modelling, according to a preprint posted on arXiv. The model offers a simpler alternative to existing architectures that rely on complex initialisation and normalisation schemes. [1] The paper, authored by Rares Dolga, proposes RotRNN as a linear recurrent neural network that leverages the mathematical properties of rotation matrices. Linear recurrent networks, including State Space Models and Linear Recurrent Units, have recently achieved state-of-the-art results on long-sequence benchmarks, but their empirical performance is not fully understood and they carry drawbacks such as intricate initialisation and normalisation procedures. [1] RotRNN addresses these issues by providing what the authors describe as a simple and efficient model with a robust normalisation procedure and a practical implementation that remains faithful to its theoretical derivation. [1] The model achieves competitive performance against leading linear recurrent models on several long-sequence modelling datasets. [1] The preprint was first submitted to the arXiv repository on 9 July 2024, with a file size of 267 KB. [1] A second version followed on 6 October 2024, sized at 1,449 KB, and a third revision was posted on 23 June 2026, at 1,446 KB. [1] arXiv, which began on 14 August 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed. [6] It hosts papers across mathematics, physics, computer science, and related fields, and as of November 2024 received roughly 24,000 submissions per month. [6] The work appears under the machine learning category on arXiv, which provides a suite of experimental community tools through its arXivLabs framework. [4] These tools, accessible via tabs on the abstract page, include the Bibliographic Explorer for navigating citation trees and the CORE Recommender for discovering related open-access papers. [5] arXivLabs was formalized in 2020 to allow third-party collaborators to build features that enhance the reading and discovery experience while adhering to values of openness, community, excellence, and user data privacy. [4]