MultiHashFormer: Hash-based Generative Language Models

A new framework called MultiHashFormer enables hash-based autoregressive language generation, outperforming standard Transformer models at multiple scales while keeping the parameter cost of vocabulary expansion constant, according to a paper posted to arXiv on June 26, 2026 [1][2]. Standard language models use embedding matrices whose size grows linearly with the vocabulary, a design that inflates the parameter count as models scale [2]. Prior efforts to apply hashing for parameter efficiency were limited to encoder-only architectures because many-to-one hash collisions broke the causal chain required for autoregressive generation [2]. MultiHashFormer addresses that constraint by assigning each token a unique hash signature — a short sequence of discrete hash IDs produced by multiple independent hash functions — and then compressing that signature into a single latent vector via a Hash Encoder before passing it to a Transformer decoder [1][2]. A Hash Decoder subsequently generates the hash signature of the next token, which is mapped back to text [1][2]. The authors evaluated MultiHashFormer at the 100-million, 1-billion, and 3-billion parameter scales and report that it consistently outperforms standard Transformer language models across multiple benchmarks [1][2]. The architecture also handles multilingual vocabulary expansion without any modifications, maintaining a constant parameter footprint as new languages are added [1][2]. Transformers, first proposed in the 2017 paper “Attention Is All You Need,” have become the dominant architecture for large language models, with variants grouped into encoder-only, decoder-only, and encoder-decoder designs [5]. The decoder-only configuration underpins many of the largest generative models, including Google’s PaLM, a 540-billion-parameter dense model, and the Gemini family of multimodal models that succeeded LaMDA and PaLM 2 [3][6]. Google DeepMind’s Gemma series, which shares technology with Gemini, has progressed through several iterations, with the open-source Gemma 4 released in April 2026 [4]. The MultiHashFormer paper appeared on arXiv, the open-access e-print repository that hosts preprints across physics, computer science, mathematics, and related fields [10]. As of November 2024, arXiv was receiving roughly 24,000 new articles per month and had surpassed two million total articles by the end of 2021 [10]. The repository is moderated but does not conduct peer review [10].