MMDiff: Extending Diffusion Transformers for Multi-Modal Generation

A new framework called MMDiff repurposes frozen diffusion transformers to generate images alongside dense perceptual outputs such as segmentation maps, according to a paper submitted to arXiv on 15 Jun 2026 [1][2]. The method uses lightweight decoder heads and multi-timestep feature fusion to extract perceptual information normally discarded during the denoising process [2]. The paper, hosted on the open-access repository arXiv, details how MMDiff transforms a static diffusion transformer into a multi-modal generative system without retraining the backbone model [1][2]. The authors report that perceptual information is temporally distributed along the denoising trajectory, making single-timestep extraction insufficient [2]. By applying multi-timestep feature fusion with spatially varying aggregation weights, the framework improved semantic segmentation results by up to 28.7% mIoU over single-timestep baselines [2]. The researchers also adopted concept-driven attention extraction to provide interpretable spatial guidance [2]. Frozen diffusion features proved competitive with and complementary to state-of-the-art encoders such as DINOv3 [2]. Training only the lightweight decoder heads on the frozen backbone yielded strong performance across semantic segmentation, salient object detection, and depth estimation [2]. The framework further enables synthetic data generation at scale, a capability the authors highlight as a practical output of the system [2]. arXiv, which began on August 14, 1991, serves as a non-peer-reviewed repository for preprints across disciplines including computer science, physics, and mathematics [6]. As of November 2024, the platform received approximately 24,000 submissions per month and had surpassed two million total articles by the end of 2021 [6]. The MMDiff paper appears with the standard arXiv abstract page tools, including the arXivLabs framework, a community collaboration space that hosts experimental features such as bibliographic explorers and code finders [4][5]. arXivLabs was formalized in 2020 to allow third-party developers to build tools that integrate directly with the repository, under guidelines that enforce user data privacy and alignment with arXiv’s values of openness and community [4].