Self-Generated Error Training for Token Editing in Diffusion Language Models

Researchers have proposed a new training method for token-editing mechanisms in diffusion language models, designed to improve accuracy while reducing the intensity of edits needed during text generation, according to a preprint posted on arXiv [1]. The method addresses a specific mismatch in how the LLaDA2.1 model's token-to-token (T2T) editor is trained versus how it operates during actual use. The standard recipe trains the editor on random vocabulary corruptions, but at inference time the model encounters its own fluent, high-confidence draft errors, a discrepancy that can lead to failures [1, 2]. The new approach, called self-generated T2T, performs a no-gradient draft pass, fills masked positions with predicted tokens, and then supervises recovery in a second pass under these self-generated corruptions [1, 2]. Diffusion models, which underpin this work, are a class of generative models that learn to reverse a process of adding noise to data. While they have seen widespread commercial use in image generation through systems like Stable Diffusion and DALL-E, their application to natural language processing tasks such as text generation is a more recent development [3]. The LLaDA2.1 model uses a block-diffusion decoding process, where the T2T editor can revise tokens that have already been committed, a capability the researchers sought to refine [1, 2]. The update was implemented as a short LoRA continued-pretraining pass on the LLaDA2.1-mini variant and evaluated on several benchmarks under the official Q-Mode T2T procedure with unchanged inference parameters [1, 2]. The results showed that the method generally improves accuracy while reducing T2T edit intensity. It specifically mitigated failure modes such as final-digit transcription errors occurring after otherwise correct reasoning, and excessive self-correction before short factual answers [1, 2]. The preprint was submitted to arXiv on June 15, 2026. arXiv, an open-access repository for electronic preprints, hosts papers across physics, mathematics, computer science, and related fields, and is a primary venue for sharing machine learning research before peer review [7]. Large language models, the broader category to which diffusion language models belong, are machine learning models with many parameters trained on vast amounts of text for natural language processing tasks [9].