Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions

A new scheduling method for large language model inference replaces single-point output-length predictions with a probability distribution, aiming to better handle the inherent uncertainty of text generation, according to a paper posted on arXiv [1]. The method, called Tail Inflated Expectation (TIE), addresses a limitation in current scheduling systems that rely on the shortest-job-first principle to reduce head-of-line blocking. Existing schedulers typically predict a single output length for each request, an approach the authors argue does not reflect the stochastic decoding process where length is determined by when an end-of-sequence token is sampled [1]. The paper, authored by Haoyu Zheng, was submitted on 1 April 2026 and revised on 25 May 2026 [1]. Large language models, which are neural networks trained on vast text corpora, underpin modern chatbots and text-generation tools [3]. Their inference process is computationally intensive, and efficient scheduling is critical for both interactive and batch workloads. The TIE scheduler instead fits each request's output length to a heavy-tailed log-t distribution, then adjusts the distribution's expectation using tail probabilities to account for the risk that a request will generate a long output [1]. The researchers compared TIE against three baselines. For online inference, TIE reduced per-token latency by a factor of 2.31. For offline data generation, it improved throughput by a factor of 1.42 [1]. The first version of the paper was 839 KB, and the revised version was 825 KB [1]. The work draws on concepts of decision-making under uncertainty, a theme also central to behavioral economics, which studies how psychological factors cause decisions to deviate from traditional rational models [5]. By modeling the risk of long outputs explicitly, the scheduler makes a trade-off between average performance and worst-case delays. The paper does not include external quotes or third-party commentary [1].