Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

A new Transformer architecture called Flexformer learns its own attention kernels directly from data, sidestepping the quadratic complexity that bottlenecks standard models on long sequences, according to a preprint posted June 26, 2026 [1][2]. Standard Transformer models use an attention mechanism that scales quadratically with sequence length, which limits their use on long documents or high-resolution inputs [1][2]. Kernel-based linear attention reduces that cost but has historically relied on fixed or only weakly tunable kernels, capping what the model can express [2]. Flexformer builds on random Fourier feature-based linear attention and treats the spectral frequencies inside those features as trainable parameters, allowing the model to learn a broad family of attention kernels in a fully data-driven way [2]. The authors develop both stationary and nonstationary variants; the nonstationary version offers strictly greater expressiveness [2]. In experiments covering language modeling and sequence classification, Flexformer consistently outperformed baseline linear-attention models [2]. The team also showed that a Flexformer can be distilled from a pretrained softmax Transformer and recover softmax-like attention patterns, while retaining the efficiency of linear attention [2]. The learned kernels transferred across domains, suggesting the model captures reusable structure rather than overfitting to a single task [2]. Kernel methods have a long history in machine learning. The 2015 work by Mairal and colleagues on universal acceleration for first-order optimization demonstrated how well-chosen auxiliary problems can speed up convex minimization, a principle that echoes the idea of learning better kernel representations inside neural networks [3]. Flexformer applies that spirit to the attention mechanism itself, making the kernel an object of optimization rather than a hand-picked function. The preprint was submitted to arXiv on 26 June 2026 and is hosted under the Machine Learning category [1]. The paper’s code and data links appear on the abstract page through services such as CatalyzeX and Hugging Face, though the repository contents were not detailed in the available metadata [1][4][5]. The work arrives as the research community continues to search for architectures that can handle the growing length of real-world sequences without sacrificing the representational power that made Transformers dominant.