Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

A new method called Top-Theta Attention can dramatically reduce the memory footprint of transformer models during inference without any retraining, according to a paper posted on arXiv. The technique uses static, per-head thresholds to selectively retain only the most significant attention elements. The approach, detailed in a preprint by Konstantin Berestizshevsky, introduces a training-free sparsification strategy for the attention mechanism central to large language models [1]. Unlike existing top-k attention methods, Top-Theta Attention calibrates a fixed threshold for each attention head. This allows the model to dynamically retain a constant number of important elements per row based on content, rather than a pre-set number, making it robust across different types of data [2]. The paper reports that this method achieves a 3-10x reduction in V-cache usage and requires up to 10x fewer attention elements during inference, while degrading accuracy by no more than 1% [1, 2]. The research addresses a core computational bottleneck in transformer models, which are a type of machine learning model with many parameters trained on vast amounts of text for language generation tasks [8]. The preprint was submitted to arXiv, an open-access repository for scientific e-prints that is not peer-reviewed but serves as a primary dissemination platform in fields like computer science and physics [6]. The repository, which began in 1991, now receives about 24,000 new articles per month [6]. The paper has been revised twice since its initial submission in February 2025, with the latest version posted in June 2026 [1]. The authors also introduce compensation techniques to maintain accuracy even under aggressive sparsification, positioning thresholding as a practical alternative to existing methods [2]. The work is presented as a principled approach to making large transformer models more efficient for real-world deployment, where computational and memory constraints are critical.