HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

A research team has introduced HARD-KV, a framework designed to resolve a fundamental mismatch between dynamic compression algorithms and the rigid memory requirements of modern inference engines for long-context large language models [1]. The framework, detailed in a preprint, targets a conflict where head-adaptive compression algorithms dynamically fluctuate memory budgets for accuracy, while inference engines like vLLM demand static memory patterns to leverage CUDA Graphs and PagedAttention [1][2]. HARD-KV bridges this divide through a Cascade Cache hierarchy that manages the token lifecycle across dense, sparse, and condensed tiers [2]. A Logits Calibration mechanism normalizes diverse importance metrics into a unified probability space, enabling consistent Top-$p$ budgeting across heterogeneous heads [2]. To address system efficiency, the framework rewrites fragmented, dynamic indices into contiguous physical layouts compatible with high-performance engines [2]. Experiments on math-reasoning benchmarks, including AIME and U-Math, show HARD-KV achieves up to 2$\times$ throughput improvement over static baselines while maintaining high-fidelity generation in 10k+ token scenarios [1][2]. The code is publicly available [2]. The work addresses a growing challenge in deploying large language models for extended contexts, where the computational cost of key-value caches scales with sequence length [1]. The Cascade Cache hierarchy allows the system to retain critical information in dense form while compressing less important tokens, reducing memory footprint without sacrificing output quality [2]. The Logits Calibration step is central to the approach, as it reconciles the varying importance scores produced by different attention heads into a single, actionable probability distribution [2]. This enables the system to apply a uniform Top-$p$ threshold across all heads, a process previously hindered by incompatible metrics [2]. The system-level index rewriting further ensures that the theoretical gains in memory efficiency translate to practical speedups on hardware optimized for contiguous memory access [2].