Attention Sinks in Diffusion Transformers: A Causal Analysis

A new causal analysis finds that removing attention sinks in diffusion transformers does not impair text-image alignment or preference proxies in most cases, challenging assumptions about their functional importance in these models. The study, conducted by Fangzheng Wu and submitted to arXiv on May 10, 2026, examined attention sinks—tokens that receive disproportionate attention mass—in text-to-image diffusion transformers [1]. While these sinks are assumed to be functionally important in autoregressive language models, their role in diffusion transformers had remained unclear [2]. The research dynamically identified dominant attention recipients per timestep and suppressed them through paired, training-free interventions on the score and value paths [2]. Across 553 GenEval prompts on Stable Diffusion 3, with additional corroboration using SDXL, removing these sinks at an intervention strength of k=1 did not degrade text-image alignment as measured by CLIP-T, nor did it harm preference proxies such as ImageReward and HPS-v2 [2]. Only under stronger interventions, where k reached 10 or higher, did HPS-v2 show a metric-dependent boundary, while CLIP-T remained robust throughout the testing [2]. The perceptual shifts induced by suppression were sink-specific and approximately six times larger than those caused by equal-budget random masking [2]. This finding reveals an empirical dissociation between trajectory-level perturbation and semantic alignment in diffusion transformers [2]. The paper, titled "Attention Sinks in Diffusion Transformers: A Causal Analysis," was last revised on June 16, 2026 [1]. arXiv, where the paper was posted, is an open-access repository of electronic preprints approved for posting after moderation but not peer reviewed [7]. It consists of scientific papers in fields including computer science, mathematics, and physics, and has been operating since August 14, 1991 [7]. As of November 2024, the submission rate was approximately 24,000 articles per month [7]. The repository passed the two-million-article milestone by the end of 2021 [7]. The research contributes to ongoing work in understanding how transformer-based models allocate attention internally. Large language models, a related class of machine learning models, are trained with self-supervised learning on vast amounts of text and contain many parameters [9]. The code for the attention sink analysis has been made available on GitHub [2].