Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

A new study finds that GPUs and emerging AI accelerators exhibit distinct, phase-dependent strengths when running large language model inference, with neither platform dominating across all stages of the process. The research, submitted on 14 June 2026, evaluates inference performance for the Llama2-7B model across GPUs and accelerators like GroqRack by separately measuring the Prefill and Decode phases [1][2]. These phases are tracked by time to first token (TTFT) and time per output token (TPOT), respectively [1]. The paper notes that large language models are increasingly deployed in latency- and cost-sensitive settings, making inference efficiency a central systems challenge [1][2]. The evaluation shows GPUs consistently excel in the compute-intensive Prefill phase [1][2]. In contrast, GroqRack achieves significantly lower TPOT during the Decode phase, though it does not currently support batching [1][2]. GPUs regain an advantage in Decode throughput as batch size increases [1][2]. The authors further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized [1][2]. The paper arrives as the market for AI hardware faces intense scrutiny. Chinese firm DeepSeek, which develops large language models, reported training its V3 model for US$6 million, far less than the US$100 million cost for OpenAI's GPT-4 in 2023, using approximately one-tenth the computing power consumed by Meta's comparable model, Llama 3.1 [7]. DeepSeek's success sent shock waves through the industry and contributed to a US$600 billion single-day market value loss for Nvidia, the largest single-company decline in U.S. stock market history [7]. Other major model families, such as Alibaba Cloud's Qwen, are distributed under free and open-source licenses like Apache 2.0 [9]. The Hugging Face Hub has become a central repository for such models, allowing researchers to link papers to models, datasets, and interactive demos [4]. Hugging Face and arXiv have also collaborated to embed demos directly alongside papers on arXiv abstract pages [5].