HAARES Half-Split Residual Basis Routing for Deep Transformers

A new lightweight routing method for deep transformers, called HAARES, shows performance gains in 48-layer models but its benefits are depth-dependent, according to a preprint posted on arXiv [1]. The method, detailed in a paper by Kehan Wang, introduces a residual basis router that retains the cumulative block source and adds a single half-split detail basis [1]. This detail basis is computed as the difference between the first-half and second-half residual updates, is RMS-matched, and is updated online, exposing coarse intra-block trajectory information without requiring dense sublayer-level routing [1][2]. The approach is designed to be FLOP-light, though the authors note it is not wall-clock-free, adding memory and routing overhead [2]. Its relative arithmetic cost is amortized as model width grows, and earlier convergence can reduce time-to-target [2]. Empirical results across OpenWebText, cross-domain character-level benchmarks, and BPE-tokenized OpenWebText reveal a depth-dependent pattern: gains are small or mixed at shallow depth and most reliable in 48-layer models [1][2]. In the 201M-parameter, 48-layer setting, HAARES improved over Block AttnRes across all three seeds, and a 453M-parameter, two-seed probe showed the same direction [1][2]. Ablation studies ruled out source duplication, random signed details, fixed detail-source biases, or block-count changes alone as explanations for the improvement [2]. The paper was submitted to arXiv on June 4, 2026, as version one with a file size of 110 KB, and revised on June 17, 2026, with version two at 2,230 KB [1]. arXiv, which began on August 14, 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed, and it surpassed two million articles by the end of 2021 [6]. The repository hosts work across mathematics, physics, computer science, and related fields, with a submission rate of about 24,000 articles per month as of November 2024 [6].