Stochastic Sparse Attention for Memory-Bound Inference

Researchers have proposed new methods to improve the efficiency of long-context Large Language Model (LLM) inference, achieving significant speedups in decode-step attention-kernel and end-to-end decode-latency.

Two recent studies published on arXiv^[1][2] present innovative approaches to enhance the performance of LLMs. The first study introduces Stochastic Additive No-mulT Attention (SANTA), a method that sparsifies value-cache access by sampling a subset of indices from the post-softmax distribution, resulting in an unbiased estimator of the post-softmax value aggregation^[1]. This technique achieves up to 1.5x decode-step attention-kernel speedup and 1.25x end-to-end decode-latency speedup in batched long-context generation. The second study proposes SparDA, a decoupled sparse attention architecture, which introduces a fourth per-layer projection called the Forecast to predict KV blocks needed by the next layer^[2]. SparDA adds less than 0.5% parameters and trains only the Forecast projections, delivering up to 1.25x prefill speedup and 1.7x decode speedup over the sparse-attention offload baseline. Both methods are complementary to existing techniques such as upstream quantization, low-rank projection, KV-cache compression, and KV-cache selection methods.