Approaching I/O-optimality for Approximate Attention

Researchers have developed a new technique for computing the attention mechanism in large language models that approaches theoretical I/O-optimality, reducing data transfers between fast and slow memory to an almost-linear cost relative to input size [1]. The work revisits the I/O complexity of computing the attention matrix, a core operation in transformer-based models. The goal is to compute the matrix \(A=\text{softmax}(Q K ^{\top}/\sqrt{d}) V\) from query, key, and value matrices \(Q,K,V\in\mathbb{R}^{n\times d}\) while minimizing data movement between fast memory of size \(M\) and slower storage [1]. Existing methods, including the widely used FlashAttention and its variants, incur an I/O cost that scales quadratically with the sequence length \(n\). A trivial lower bound requires only \(\Omega(nd)\) I/O operations to read the inputs and write the output [1]. The new technique achieves an I/O cost that depends almost-linearly on \(n\) across most parameter regimes. It accomplishes this by developing I/O-efficient algorithms inspired by the approximate attention framework recently introduced by Alman and Song [1]. The authors also prove corresponding lower bounds for each parameter regime, demonstrating that their algorithms are close to I/O-optimal [1]. This advancement addresses a critical computational bottleneck. The travelling salesman problem, an NP-hard optimization challenge studied since 1930, similarly grapples with finding efficient routes through data points and is used as a benchmark for optimization methods [4]. While the TSP has exact algorithms for tens of thousands of cities and approximations for millions, the attention computation in language models faces a comparable scaling challenge where quadratic costs become prohibitive for long sequences [4]. Multi-objective optimization provides another relevant lens, as it deals with trade-offs between conflicting goals, such as maximizing performance while minimizing computational resource consumption [5]. The new attention algorithm effectively navigates a trade-off between computational accuracy and I/O efficiency, finding a solution that is nondominated in the sense that I/O cost cannot be improved without degrading other objectives [5]. The paper was submitted on 22 May 2026 and is available on the arXiv preprint server [1].