DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels

A team of researchers has introduced DICE, a series of diffusion large language models tailored for CUDA kernel generation that outperforms both autoregressive and diffusion models of comparable scale on the KernelBench benchmark, according to a paper posted to arXiv [1]. The models, spanning 1.7B, 4B, and 8B parameters, were built to address two obstacles that have limited the application of diffusion LLMs to CUDA kernel code: the high specialization required and a severe shortage of high-quality training data [1][2]. To overcome the data scarcity, the researchers constructed CuKe, an augmented supervised fine-tuning dataset optimized for high-performance CUDA kernels [3][4]. The dataset contains a large number of high-performance kernel examples, providing the foundation for task-specific supervision [5]. On top of CuKe, the team built a bi-phase curated reinforcement learning framework called BiC-RL [1][6]. The framework operates in two sequential stages: a CUDA kernel infilling stage, where the model learns to complete partial kernel implementations, followed by an end-to-end CUDA kernel generation stage [3][5]. During training, data is scheduled hierarchically, progressing from fundamental operations to complex model structures [3]. This progressive paradigm ensures the model masters core kernel implementation before transitioning to full generation [3]. The resulting DICE models represent what the authors describe as the first specialized diffusion LLMs designed for CUDA kernel generation [4][5]. In experiments on KernelBench, the DICE series established a new state-of-the-art, significantly outperforming both autoregressive LLMs and other diffusion LLMs of comparable scale [1][2]. The paper notes that recent studies have demonstrated the superior data efficiency of diffusion LLMs, especially under data-bound conditions, making them uniquely positioned for tasks where high-quality kernel data is scarce [3]. The work was submitted by Haolei Bai and collaborators, with the initial preprint posted on February 12, 2026, and a revised version uploaded on June 16, 2026 [1].