Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

A new adaptive framework called Nightjar can dynamically tune speculative decoding for large language model serving, boosting throughput by up to 14.76 percent and cutting latency by up to 20.18 percent under fluctuating request loads, according to research published on arXiv [1][2]. Speculative decoding accelerates large language model inference by generating multiple draft tokens and verifying them in parallel. The technique improves throughput when systems are memory-bound under light load, but it degrades performance in compute-bound, high-load environments because the verification step adds overhead and the draft model consumes GPU memory that could otherwise hold key-value cache for larger batches [1][2]. Existing speculative decoding methods rely on fixed draft lengths and cannot adapt to workload changes or determine when speculation should stop. The cost of restarting speculative inference has also remained unquantified [2]. To address these limitations, researchers led by Li Rui propose Nightjar, a resource-aware adaptive speculative framework [1]. Nightjar dynamically selects the optimal speculative length for different batch sizes as request loads shift. A multi-armed bandit planner within the framework decides when speculation ceases to be beneficial and proactively disables it. During the disabled phase, the draft model is offloaded to the CPU only when GPU memory is under pressure, reclaiming space for the KV cache and enabling larger batch sizes [2]. In experiments conducted under dynamic request arrival rates for real-time serving scenarios, Nightjar delivered up to 14.76 percent higher throughput than standard speculative decoding and up to 20.18 percent lower latency in the main benchmark suite [1][2]. The paper was initially submitted on December 27, 2025, and revised through five versions, with the latest dated June 15, 2026. The submission history shows file sizes ranging from 1,944 KB in the first version to 887 KB in the fifth [1]. The work arrives as the machine learning community continues to explore efficiency gains for large-scale model deployment. Broader research on transfer learning and dataset consolidation, such as efforts to combine the OC20 and OC22 catalysis datasets, has shown that computational strategies can be transferred across related tasks to improve model performance [4]. While Nightjar focuses on inference-time adaptation rather than training, both lines of work reflect an ongoing push to extract more capability from constrained compute resources.