Taming Curvature: Architecture Warm-Up for Stable Transformer Training

Researchers have proposed a novel 'architecture warm-up' method to stabilize the training of billion-parameter Transformers, a process often plagued by loss spikes and divergence.

The Edge of Stability (EoS) theory offers a tool to understand optimization method stability via curvature, but its application is hindered by complex curvature estimation^[1]. A new online estimator for the largest Hessian eigenvalue addresses this issue, making per-iteration curvature tracking feasible at scale. Experiments show training instabilities coincide with surges in preconditioned curvature, which grows with depth^[1]. The proposed architecture warm-up method progressively increases network depth to control the preconditioned Hessian, stabilizing training. Meanwhile, a separate research effort introduced NeuronFabric, a software reference architecture for on-chip transformer training with local Adam updates, intended for future FPGA and ASIC implementations^[2]. A C# prototype of NeuronFabric implements forward pass, backpropagation, and Adam optimization without external frameworks. Evaluations on a 334K-parameter autoregressive transformer trained on the Shakespeare corpus showed the BF16W configuration achieving an evaluation loss of 1.5426 after 80K samples^[2].