Sumi: Open Uniform Diffusion Language Model from Scratch

A team of researchers has released Sumi, a fully open 7-billion-parameter uniform diffusion language model pretrained from scratch on 1.5 trillion tokens, marking the first native UDLM trained at both large parameter and token scale [1]. The model, whose name means "ink" in Japanese, was introduced in a paper submitted in 2026 by authors Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, and Jun Suzuki [1][4]. Unlike autoregressive models that generate text token by token, uniform diffusion language models permit any token to be updated at any step, which in principle enables more flexible generation [2]. Until now, no UDLM had been pretrained from scratch at this scale, leaving the community without a clean reference point for studying scaling behavior, generation dynamics, and controllability [3]. Sumi runs full bidirectional attention and denoises a canvas of randomly corrupted tokens [4]. The model weights, intermediate checkpoints, and full training recipe have been released, including a complete specification of the data mixture over publicly available corpora [1][5]. The model is available on Hugging Face under the repository tohoku-nlp/sumi-7b and requires a custom model class, meaning users must set trust_remote_code=True to load it in the Transformers library [4]. The recommended Transformers version is 5.8.1 [4]. On benchmarks, Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding tasks [1][3]. It underperforms on commonsense benchmarks, a gap the authors attribute to an education- and code-heavy data mixture [2][5]. The release provides a foundation for studying native uniform diffusion at scale and aims to catalyze work on aspects of the approach that remain poorly understood [1]. The release comes as diffusion models gain traction as alternatives to autoregressive architectures. Large language models are typically trained with self-supervised learning on vast amounts of text [11]. Other recent work has explored diffusion for specialized domains such as quantum circuit generation, though a structured review found no system reporting end-to-end evaluation on quantum hardware [6]. The Sumi release adds a general-purpose language model to the growing landscape of open-weight models, following a trend in which organizations release model parameters under open licenses to enable community research and reproduction [10].