Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

A new optimizer wrapper called Hyperball aims to sustain the pretraining speed gains of matrix-based optimizers such as Muon as language models scale up, according to a paper posted to arXiv on June 15 [1][2]. The method forces weight matrices and their updates to maintain fixed Frobenius norms, countering a known pattern where Muon’s advantage over AdamW diminishes with larger models and datasets [2]. The work, titled “Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization,” addresses a specific bottleneck in large language model training [1]. Matrix-based optimizers like Muon can substantially accelerate pretraining, but researchers observed that their edge over the widely used AdamW shrinks when standard constant decoupled weight decay is applied at greater model sizes and data scales [2]. Hyperball operates as a wrapper around a base optimizer such as Adam or Muon, setting the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants [2]. On Qwen3-style models with up to 1.2 billion parameters, Muon Hyperball delivered a 20–30% token-equivalent speedup over weight decay baselines [2]. The paper also reports that Hyperball improves learning rate transfer across different model widths and depths compared to decoupled weight decay [2]. The approach is grounded in prior theory showing that training with weight decay leads to an equilibrium weight norm that depends solely on the training hyperparameters [2]. Through this mechanism, weight decay effectively governs the angular learning rate—how fast the direction of the weight matrix changes [2]. Large language models, which underpin modern natural language processing systems, are trained with self-supervised learning on vast text corpora and can contain billions of parameters [8]. The preprint appeared on arXiv, an open-access repository that hosts e-prints across physics, computer science, and related fields without peer review [6]. As of late 2024, the repository was receiving roughly 24,000 submissions per month [6]. The paper’s abstract page includes links to experimental community tools developed under the arXivLabs framework, such as the Bibliographic Explorer and CORE Recommender, which help readers navigate citation trees and discover related open-access research [4][5].