Service-Induced Congestion in Memory-Constrained LLM Serving

Researchers have identified service-induced congestion as a major issue in large language model (LLM) serving, leading to throughput losses as large as 50%^[1].

In LLM serving, each request accumulates persistent GPU memory as its key-value cache grows with every generated token^[1]. When memory capacity is exceeded, systems evict active requests, discarding cached state and restarting them later. This wastes computation and reduces throughput. A study on arxiv.org^[1] developed a discrete-time dynamical model of memory-constrained LLM inference, capturing admission, memory growth, and eviction under continuous batching. The study found that the system admits both eviction-free fixed points and limit cycles with evictions in the saturated-input regime. For homogeneous workloads, the eviction-free equilibrium is unstable, and the system converges to a unique worst-case limit cycle outside a Lebesgue-measure-zero exact-capture set. Meanwhile, another study on arxiv.org^[2] proposed PolyKV, a layer-wise KV cache optimization framework that improves KV cache compression for long-context LLM inference. PolyKV routes each layer to a suitable KV compression policy based on layer-level signals and assigns non-uniform budgets under a fixed total budget. Experiments on LLaMA-3.1-8B and Qwen3-8B showed that PolyKV recovered 54.5% and 25.7% of the LongBench performance gap between the strongest single-policy baseline and FullKV, respectively^[2].