DMuon: Efficient Distributed Muon Training with Near-Adam Overhead

A new open-source distributed implementation of the Muon optimizer, called DMuon, reduces per-step training latency to levels comparable with the widely used AdamW algorithm, according to a preprint posted to arXiv on June 25 [1][2]. The work addresses a bottleneck that previously made matrix-orthogonalization-based optimizers prohibitively expensive for large-scale distributed training. Matrix-level optimizers such as Muon couple entire weight matrices during updates, requiring costly Newton-Schulz iterations that are poorly served by infrastructure built for element-wise methods [1][2]. In standard setups, vanilla Muon implementations add more than twice the cost of the forward and backward passes combined [1][2]. The DMuon project closes that gap by delivering a drop-in module that requires no framework-level modifications [1][2]. Across embodied foundation model and large language model training workloads, DMuon achieves a 1.48x to 3.01x speedup in end-to-end step time and a 6.85x to 163.00x speedup in optimizer-step time, bringing per-step latency to near-AdamW levels [1][2]. The preprint was submitted to the Distributed, Parallel, and Cluster Computing section of arXiv, an open-access repository that hosts e-prints across physics, computer science, and related fields [1][6]. arXiv, which began in 1991, now receives roughly 24,000 submissions per month and surpassed two million articles at the end of 2021 [6]. Papers on the platform are moderated but not peer-reviewed, a distinction that places the DMuon results in the category of preliminary findings awaiting formal publication [6]. Matrix-orthogonalization-based optimizers have drawn attention because they exhibit strong convergence behavior across diverse modern deep learning workloads, offering an alternative to conventional element-wise optimization as model architectures grow in scale and heterogeneity [1][2]. The DMuon authors note that the implementation is open-source and designed to integrate into existing training pipelines without altering the underlying framework, a feature that could lower the barrier for research groups exploring matrix-aware update rules [1][2]. The speedup figures reported span a wide range, reflecting differences in model size, parallelism strategy, and hardware configuration, though the paper does not detail those variables in its abstract [1][2].