Efficient On-Device Diffusion LLM Inference with Mobile NPU

A new inference framework called llada.cpp reduces the generation latency of a diffusion large language model on smartphones by 17 to 42 times compared to a CPU-only baseline, according to a preprint posted to arXiv on June 11, 2026 [1][2]. The framework targets diffusion large language models, or dLLMs, which produce text by denoising multiple tokens in parallel rather than generating one token at a time [1][2]. That parallel approach makes dLLMs appealing for mobile applications where users expect fast responses, but the repeated denoising steps still impose a heavy computational load on smartphones [1][2]. Mobile neural processing units, or NPUs, can deliver high-throughput dense matrix computation, yet three obstacles have blocked efficient use of those chips for dLLM inference: shrinking per-block workloads as tokens are committed, complications with key-value cache reuse when tokens are revised, and costly data remapping caused by limited NPU-visible address space [1][2]. The authors of the paper introduce three techniques to align block-wise dLLM inference with the execution characteristics of mobile NPUs [1][2]. Multi-Block Speculative Decoding fills the thinning workload in late-stage current-block decoding with speculative future-block tokens [1][2]. Dual-Path Progressive Revision keeps committed tokens revisable until they stabilize and refreshes unstable tokens through a CPU-side path without stalling dense NPU execution [1][2]. Swap-Optimized Memory Runtime compacts NPU-visible address layouts and overlaps data staging with NPU computation to reduce remapping and transfer overheads [1][2]. When evaluated on the LLaDA-8B model, the end-to-end llada.cpp framework cut generation latency by a factor of 17 to 42 compared with a CPU baseline that reuses a prefix key-value cache, while preserving generation quality [1][2]. The preprint appears on arXiv, an open-access repository that hosts electronic preprints across physics, computer science, and related fields and that surpassed two million articles by the end of 2021 [7]. The work arrives as on-device inference for large language models — machine learning systems with many parameters trained on vast text corpora — remains an active area of research and engineering [9].