Depth-Attention: Cross-Layer Value Mixing for Language Models

A new method called Depth-Attention lets Transformer language models selectively reuse information from earlier layers without adding parameters or extra inference memory, according to research posted to arXiv on June 3. The technique operates inside the attention module itself, mixing value states across depths before standard self-attention runs. In a standard Transformer, each layer adds its output to a residual stream, meaning later layers cannot selectively pull representations from earlier ones [1]. Recent cross-layer approaches have tried to improve this flow, but they typically operate on hidden states outside the attention mechanism, requiring additional persistent state beyond the key-value (KV) cache at inference [2]. That extra memory cost has become more significant as modern large language models adopt cache-compression techniques such as grouped-query and multi-head latent attention [2]. Depth-Attention addresses the problem differently. Before a layer performs self-attention across the sequence, its query attends over the keys of earlier layers at the same token position and mixes their values into an updated value state [3]. The layer then runs standard causal self-attention on this depth-mixed value, leaving queries, keys, and the causal mask unchanged [3]. Because the method reuses the standard attention queries, keys, and value-cache slots — storing depth-mixed values in place of the originals — it adds no parameters and introduces no persistent inference state beyond the standard KV cache [4]. The cache size remains identical to a vanilla decoder, which is smaller than what hidden-state-based cross-layer methods require [4]. The researchers evaluated Depth-Attention on Qwen3-style decoders at 1.5B and 3B parameters [1]. It attained the lowest perplexity and the highest average downstream accuracy, improving over the vanilla Transformer by up to 2.3 accuracy points [1]. It also surpassed strong cross-layer baselines — including mHC, Attention Residuals, and DenseFormer — in both perplexity and average accuracy [3]. The computational overhead is minimal: the method adds under 0.01% extra arithmetic FLOPs [1]. Gains held across model sizes from 360M to 3B parameters and extended to looped Transformer architectures [2]. The work follows a broader line of research into depth-wise information mixing. A related approach, mixture-of-depths attention (MoDA), was recently proposed to address what its authors call the “information dilution problem” in deep LLMs [5]. MoDA allows each attention head to jointly attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers, with a hardware-efficient fused kernel that reaches 97.3% of FlashAttention-2 efficiency at 64K sequence length [5]. Depth-Attention occupies a distinct point in the design space by placing the cross-layer interaction at the value-state level rather than storing additional histories of full hidden states [4]. Transformers, which rely on self-attention mechanisms rather than recurrence, have become the dominant architecture for many sequence-processing tasks in recent years [7]. The Depth-Attention paper appears on arXiv as a preprint and has not yet been peer-reviewed [1].