Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

A new method called Grouped Query Experts (GQE) applies mixture-of-experts routing to the attention mechanism of Transformer models, selectively activating only a subset of query heads per token while leaving the key-value pathway unchanged, researchers report in a paper posted to arXiv [1]. Self-attention is a core driver of Transformer performance but becomes the dominant computational cost at long context lengths because pairwise token interactions scale quadratically with sequence length [1][2]. Standard dense attention applies the same set of attention heads to every token regardless of that token's difficulty or information content, a uniform activation pattern that wastes compute as sequences grow longer [1][3]. The Transformer architecture, introduced in the 2017 paper "Attention Is All You Need," underpins most modern large language models and has been widely adopted across natural language processing, computer vision, and other domains [6]. The proposed GQE method builds on grouped-query attention, or GQA, which reduces key-value cache memory and bandwidth by sharing keys and values across groups of query heads [3][4]. GQA keeps all query heads active, however. GQE keeps the GQA key-value path unchanged and instead routes each token to a sparse subset of query-head experts within each GQA group, resizing the output projection accordingly [1][5]. A router selects k query-head experts per token, and the design includes an always-on shared head to anchor the routing signal [3][5]. On a fixed 30-billion-token training budget at the 250-million-parameter scale, GQE matched the all-active GQA baseline in downstream accuracy while activating half of the routed query-head experts per token [1][2]. In the main experimental setting, this corresponds to computing 9 of 16 total query-attention heads once the shared head is included [3][5]. The approach also achieved a 1.7–1.8× prefill speedup at long context lengths [3][4]. The researchers note that straightforward routing without renormalized scoring falls below the baseline, and the corrected method is necessary to recover full accuracy [3][5]. Conditional computation via mixture-of-experts layers has previously been applied to Transformer MLP blocks. Chinese AI firm DeepSeek, for instance, incorporated mixture-of-experts techniques to reduce training costs for its R1 model, which was trained for a reported US$6 million—far less than the estimated US$100 million cost of OpenAI's GPT-4 [7]. The GQE paper asks whether that conditional-computation idea can be moved into the attention block itself [3][4]. The work was submitted to arXiv on June 18, 2026, by Vishesh Tripathi and revised on June 23, 2026 [1].