Unifying Local Communications and Local Updates for LLM Pretraining

Researchers have introduced two new techniques, GASLoC and AdaGC, to improve the pre-training of large language models (LLMs). GASLoC is a decentralized algorithm that outperforms state-of-the-art methods in certain settings, while AdaGC is an adaptive gradient clipping scheme that enhances stability.

GASLoC, introduced in a paper submitted in 2026^[1], is a novel decentralized pre-training algorithm that allows for local optimizer steps and sparse randomized peer communication. It generalizes the notion of communication acceleration to the 'outer optimizer', making it compatible with adaptive optimizers. According to the researchers, GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for certain topologies. In experiments, GASLoC was able to obtain performance competitive with DiLoCo when utilizing multiple local steps. Meanwhile, a separate research effort proposed AdaGC, an adaptive per-tensor gradient clipping scheme designed to mitigate loss spikes in large-scale language model pretraining^[2]. AdaGC is optimizer-agnostic and introduces negligible memory overhead. Experiments on Llama-2 7B, Mixtral 8x1B, and ERNIE 10B-A1.4B demonstrated that AdaGC robustly eliminates training instabilities, improving downstream accuracy over GlobalGC by 1.32%, 1.27%, and 2.48%, respectively^[2].