GradPower: Powering Gradients for Faster Language Model Pre-Training

A team of researchers has introduced GradPower, a lightweight gradient-transformation technique that accelerates language model pre-training by applying an elementwise sign-power transformation to the gradient vector, requiring only a single-line code change and no alterations to the base optimizer's hyperparameters [1]. The method, detailed in a paper submitted to ICLR 2026, transforms a raw gradient vector by raising its absolute values to a fixed power while preserving the original sign, then feeds the result into a standard optimizer [2][4]. When integrated with AdamW, the resulting optimizer—termed AdamWPower—consistently achieves lower terminal loss across a range of architectures, including LLaMA and Qwen2MoE, at parameter scales from 66 million to 2 billion [1][5]. The authors tested the technique on the C4 and OpenWebText corpora under both cosine-decay and warmup-stable-decay learning-rate schedules, with the most significant performance gains appearing in modern mixture-of-experts models using warmup-stable-decay schedules [1][5]. The approach also integrates with other optimizers. The paper reports that combining GradPower with Muon yields further improvements, demonstrating compatibility beyond the Adam family [1][3]. The technique operates on the gradient before it reaches the optimizer, a design that leaves the optimizer's internal logic untouched. This contrasts with normalization methods commonly used in deep learning, which typically rescale activations or input data to stabilize training and reduce sensitivity to feature scales [7]. The submission history shows the paper was first posted on 30 May 2025, with revisions on 2 April 2026, 20 May 2026, and a final version on 13 June 2026 [1]. The authors—Mingze Wang, Jinbo Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, and Lei Wu—provide theoretical analyses that examine the influence of gradient noise on the transformation's mechanism [3][4]. The work was submitted under the primary area of optimization to the ICLR 2026 conference [4].