Blurry Window Attention

Researchers have introduced Blurry Window Attention (BLA), a new linear attention model designed to address the computational bottlenecks Transformer language models face when processing long sequences, according to a paper published on arXiv [1]. The Softmax Attention operation central to Transformer models suffers from quadratic complexity relative to sequence length and a continuously growing state size in the form of a KV cache, which becomes a bottleneck in long-context scenarios [1]. To overcome this, alternative architectures with linear complexity and a finite state size have been developed, including State-Space Models (SSMs), Linear Attention (LA), and Attention with Bounded-memory Control (ABC) [1]. While these linear models achieve language perplexity scores similar to Transformers, they have lagged behind on tasks requiring the retrieval or recall of specific information [1]. BLA is a novel ABC method inspired by SSMs [1]. It operates by storing a frequency window, from which a blurry KV history is reconstructed through interpolation using Dirichlet kernels [1]. The architecture can be understood as a generalization of Sliding Window Attention (SWA) depending on the Dirichlet kernel resolution, or as a special case of Gated Slot Attention (GSA) where the decay factor is implemented with Dirichlet kernels [1]. On the Multi-Query Associate Recall (MQAR) synthetic task, the state efficiency of BLA proved to be 8× better than SWA and competitive with popular linear attention models [1]. In the RegBench synthetic task, BLA and SWA were the only models among the tested linear architectures that improved their performance as the state size grew [1]. The paper details both the theory and an efficient implementation of the approach [1].