Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

Researchers have proposed new frameworks to improve the efficiency of sequential models, addressing the computational complexity bottleneck of softmax transformers in long-context scenarios.

The quadratic computational complexity of softmax transformers becomes a bottleneck in long-context scenarios, according to a paper submitted to arXiv^[1]. Linear attention model families offer a more efficient direction, but their expressivity remains limited. Previous work interleaved softmax and linear attention layers, but efficiency remained bottlenecked by softmax attention layers. A new framework, Neural Attention Search Linear (NAtS-L), applies both linear and softmax attention operations within the same layer on different tokens, automatically determining whether a token can be handled by a linear attention model or requires softmax attention. NAtS-L provides a strong yet efficient token-level hybrid architecture. Another study on arXiv^[2] introduced Exact Linear Attention (ELA), which eliminates approximation error and addresses gradient explosion and token attention dilution. ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention^[2]. Furthermore, YOLO-LAT, an extension of linear attention to vision models, attained up to 4.3x GPU inference speedup and 7.9x parameter reduction.