Selective Rotary Position Embedding

Researchers have introduced Selective Rotary Position Embedding (Selective RoPE), an input-dependent mechanism that generalizes Rotary Position Embeddings by enabling rotation at arbitrary, learnable angles for both linear and softmax transformer architectures [1]. The work, authored by Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, and Volkan Cevher, was first submitted in November 2025 and last revised in June 2026 [1]. It was accepted as a poster at ICLR 2026 [3]. The paper argues that while standard RoPE encodes positions through fixed-angle rotations, linear transformers have relied on input-dependent gating that only decays past key-value associations, lacking a rotational component [1]. The authors provide a theoretical argument that both rotation and decay are necessary in well-performing sequence models [4]. A key insight is that softmax attention implicitly performs a hidden form of these selective rotations on query-key pairs, revealing an inherent positional structure [2]. The team formalized this by equipping gated linear attention with Selective RoPE, creating a complex-valued recurrent layer. The real part of the state manages forgetting, while the imaginary part encodes positions through rotations [2]. The implementation uses the “RoPE trick” to maintain efficiency, adding what the authors describe as minimal architectural overhead [3]. Empirical validation included synthetic benchmarks for copying, state tracking, and the Multi-Query Associative Recall (MQAR) task, alongside a 370 million-parameter language model pre-training run [3]. The method improved recall, downstream accuracy, and expressivity for gated linear attention variants [4]. The researchers also demonstrated that the technique can be applied to softmax transformers, not just linear models [1]. The code implementation has been open-sourced [3].