UltraQuant: 4-bit KV Caching for Context-Heavy Agents

A new 4-bit key-value cache compression method called UltraQuant reduces time-to-first-token by up to 3.47x for context-heavy, multi-turn agent workloads, according to a paper posted to arXiv on June 18, 2026 [1][2]. The paper, titled “UltraQuant: 4-bit KV Caching for Context-Heavy Agents,” targets a specific bottleneck in large language model serving: the key-value (KV) cache. In agentic workloads, long prefixes are reused across many short turns, placing unusual pressure on memory and making it difficult to keep GPUs fully utilized [1][2]. The authors frame 4-bit KV-cache compression around these multi-round agent workloads, where task quality, cache residency, and serving throughput must be measured jointly [1][2]. UltraQuant is described as an FP4 approximation path that uses FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on AMD’s CDNA4 architecture [1][2]. The design draws on TurboQuant-style rotation and codebook quantization as a quality anchor, while using vLLM FP8 KV caching as the deployment anchor [1][2]. The paper also details practical design choices to make the 4-bit path robust, including asymmetric key/value treatment, Walsh-Hadamard rotation, QJL removal, and block-scale variants [1][2]. On a long-context, multi-turn agentic workload, UltraQuant cut the P50 time-to-first-token by 3.47x in the cache-pressured late rounds, and by 2.3x across all rounds, compared to an FP8 KV baseline [1][2]. Output throughput rose by 1.63x over the same baseline [1][2]. The work includes optimized decode-attention kernels developed specifically for AMD GPUs [1][2]. The paper was submitted to arXiv’s Machine Learning section (cs.LG). arXiv, which began on August 14, 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed [6]. As of November 2024, the repository receives about 24,000 submissions per month and has surpassed two million total articles [6]. The platform also hosts arXivLabs, a framework for community-developed tools that appear on article record pages, though the Labs program is currently pausing new proposals while the arXiv team focuses on a cloud migration [3][4].