Training-free sparse attention based on cumulative energy filtering

Researchers have proposed a new training-free sparse attention algorithm for Diffusion Transformers that uses cumulative energy filtering to boost computational efficiency without degrading accuracy, according to a paper published on arXiv [1]. The method, called DynSparse, reformulates token selection as a dual-goal optimization problem: maximizing sparsity while minimizing accuracy loss [1][2]. The authors note that widely used strategies such as Top-k and Top-p cannot satisfy both objectives at once. Top-p prioritizes accuracy, while Top-k fixes a computational budget but relaxes accuracy constraints [2]. The paper demonstrates that maintaining a fixed recall rate is sufficient for preserving accuracy, but a static threshold is suboptimal for cutting computational cost [3]. DynSparse instead calculates a dynamic threshold based on the cumulative energy of selected tokens, a value naturally contained within the Flash Attention operator [2][3]. This integration eliminates the need for extra masking computation overhead [1]. Sparse attention has become a focus in machine learning because the standard attention mechanism has quadratic time complexity relative to sequence length [4]. Many attention maps exhibit sparsity, meaning numerous values are near zero and their computation can be skipped [4]. Prior work such as SpargeAttention has pursued universal sparse and quantized attention for diverse models, including language, image, and video generation, using a two-stage online filter to predict attention maps and skip matrix multiplications [4]. In experiments on the Wan 2.2 vision generation model, DynSparse increased sparsity from 61.42 percent to 82 percent compared with the BLASST baseline, while keeping the VBench metric drop below 5 percent [1]. The improvement translated to an approximate 15 percent reduction in attention computation and a 1.61× increase in computational efficiency, which is 1.18× higher than BLASST [1]. The authors also report that DynSparse reaches 82.14 percent sparsity with acceptable accuracy, outperforming Top-k and Top-p in visual quality at the same sparsity level [2][3]. The algorithm can be combined with other acceleration techniques such as pre-computed masks, feature reuse, and quantization, and it can serve as a plug-in for existing models while remaining compatible with post-training to improve accuracy [2][3]. The work contributes to a broader effort in deep learning to make transformer architectures—which have become dominant for many sequence-processing tasks—more computationally tractable without sacrificing end-to-end performance [6][7].