StreamKL: Fast and Memory-Efficient KL Divergence for Boosting Attention Distillation

A new GPU primitive called StreamKL eliminates the quadratic memory cost of computing Kullback-Leibler divergence for attention distributions, enabling long-context model distillation on a single GPU, according to a paper posted to arXiv on June 18, 2026 [1]. Attention distillation — the process of training one attention distribution to match another by minimizing their KL divergence — is used in knowledge distillation, model compression, continual learning, and sparse-attention large language model training [1][2]. Large language models are machine learning models with many parameters, trained on vast amounts of text for natural language processing tasks [8]. Existing methods materialize both attention distributions before computing the KL reduction, incurring memory and input-output costs proportional to the product of query and key sequence lengths [1][2]. Those costs become prohibitive at long context lengths [1][2]. StreamKL derives an online formulation for the coupled two-distribution KL reduction, enabling a single one-pass forward kernel that streams query-key tiles through on-chip SRAM [1][2]. For the backward pass, StreamKL recomputes attention probabilities tile-by-tile, avoiding storage of quadratic intermediates [1][2]. The paper reports speedups of up to 43× and 14× over baseline methods in the forward and backward passes, respectively [1][2]. The extra high-bandwidth memory footprint of attention distillation is reduced from quadratic scaling to constant, allowing long-context distillation on a single GPU [1][2]. The paper appeared on arXiv, the open-access repository of electronic preprints that has served the physics, mathematics, and computer science communities since 1991 [6]. arXiv hosts papers that are moderated but not peer reviewed, and as of November 2024 receives roughly 24,000 submissions per month [6]. The StreamKL abstract page displays several arXivLabs integrations, including Bibliographic Explorer, Connected Papers, and Litmaps, which are experimental tools developed by community collaborators under a framework that arXiv formalized in 2020 [4][5]. arXivLabs projects operate under guidelines that require partners to share arXiv’s values of openness, community, excellence, and user data privacy [4]. The framework is currently on a hiatus for new proposals while the arXiv development team focuses on modernizing and migrating systems to the cloud [3].