Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descent

Researchers have shed new light on the mechanisms behind stochastic gradient descent (SGD) in deep learning, revealing a transient exploratory phase and a freezing mechanism that preferentially stabilizes flat solutions^[1].

SGD is a crucial component of deep learning, but its preference for flatter, more generalizable solutions has been unclear. A recent study analyzed SGD learning dynamics and identified a nonequilibrium mechanism governing solution selection during training. The researchers found that SGD noise reshapes the loss landscape into an effective potential that stabilizes flat solutions^[1]. Meanwhile, another study examined mini-batch stochastic steepest descent, finding that momentum enables small-batch convergence to an approximate max-margin solution through a batch-momentum trade-off^[2]. The same researchers also noted that variance reduction can recover the exact full-batch implicit bias for any batch size, albeit at a slower convergence rate. The two studies, submitted to arXiv in January and February 2026, respectively, provide new insights into the workings of SGD and its variants.