Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

A new optimizer modification called Magnitude–Direction (MD) Decoupling promises more predictable neural network training by separately controlling the magnitude and direction of weight vectors, according to a paper submitted to arXiv on 24 June 2026 [1]. The method, proposed by Alexander Hägele, targets a fundamental coupling in modern optimizers such as Adam and Muon. These algorithms treat each weight matrix as a single object, but every matrix carries two distinct quantities—a magnitude and a direction. Standard updates couple their dynamics: the directional change depends on the current magnitude, while the magnitude drifts as a byproduct of learning the direction, so neither is governed directly by the learning rate [1]. To manage this, practitioners rely on auxiliary recipes like weight decay and learning-rate warmup, which regulate the coupling only indirectly [1]. MD Decoupling factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates [2]. The model still sees a single fused weight tensor, so the split lives entirely inside the optimizer [3]. The approach echoes Weight Normalization, a 2016 reparameterization by Salimans and Kingma that decoupled weight vector length from direction to accelerate stochastic gradient descent [4]. However, the new method places the direction on a fixed sphere with a normalized update and learns the gains at their own rate, allowing the learning rate to set the angular update directly while the gains recover fine-grained scale control [5]. Across both Adam and Muon, MD Decoupling improved on well-tuned baselines and transferred the optimal learning rate across model width without retuning [1]. The technique also continued to help at scale on large Mixture-of-Experts (MoE) models [1]. The authors report that the modification effectively removes the need for weight decay and warmup [2]. The submission, weighing 3,534 KB, was posted at 15:40:26 UTC [1].