Improved Large Language Diffusion Models

A research team has introduced iLLaDA, an 8-billion-parameter language model trained from scratch using a fully bidirectional masked diffusion objective, departing from the autoregressive approach that dominates the field [1]. The model, detailed in a paper submitted to arXiv on June 24, 2026, was pre-trained on 12 trillion tokens and subsequently fine-tuned on a 25-billion-token instruction corpus for 12 epochs [1][2]. Unlike conventional large language models that generate text by predicting one token at a time in a left-to-right sequence, iLLaDA employs a diffusion process. Diffusion models, more commonly associated with image generation tools like Stable Diffusion and DALL-E, learn to reverse a process of adding noise to data, allowing them to generate new samples iteratively [3]. The researchers report that iLLaDA improves broadly across general, mathematical, and code benchmarks when compared to its predecessor, LLaDA [1]. The base version of the model, iLLaDA-Base, posted a 21.6-point improvement on the BBH benchmark and a 14.9-point gain on ARC-Challenge [1][2]. An instruction-tuned variant, iLLaDA-Instruct, showed a 14.5-point increase on the MATH dataset and a 16.5-point improvement on the HumanEval coding benchmark [1][2]. Despite its non-autoregressive training, iLLaDA remained competitive with Qwen2.5 7B, an established model of comparable scale, on several evaluations [1][2]. The paper also introduces two technical methods: variable-length generation to improve efficiency and confidence-based scoring for multiple-choice tasks [1]. Large language models are typically built on the transformer architecture and pre-trained to predict the next word in a sequence, a method known as autoregressive factorization [4]. The iLLaDA project challenges this orthodoxy by demonstrating that a diffusion-based approach, which allows the model to consider context from both directions simultaneously during training, can yield competitive results when scaled to 8 billion parameters [1][2]. The model weights and code have been made publicly available [1]. The paper appears on arXiv, an open-access repository where scientific preprints are posted after moderation but before formal peer review [6].