The Weight Norm Sets the Grokking Timescale: A Causal Delay Law

A new study proposes a causal law for the grokking timescale in neural networks, identifying the weight norm as the primary driver of the delay before generalization begins, according to research posted on arXiv [1]. Grokking describes a phenomenon where neural networks suddenly generalize to a solution long after they have perfectly memorized the training data [1]. The precise cause of this delay has been contested: some prior work reported a critical weight norm at the transition point, while other studies observed grokking without any fixed norm [2]. Researchers at arXivLabs, a framework for community-developed features on the preprint repository [4], have now intervened directly on the norm during training to settle the question [1]. Under standard training with weight decay, the network groks when its weight norm reaches a critical value, Wc. This value shows little variation across different random seeds and learning rates, with a coefficient of variation between 1 and 2 percent [1]. The critical norm also scales with the modular base following a power law [2]. To test causality, the team clamped the weight norm to a fixed multiple, rho, of Wc and held it there. The network still grokked, but the delay followed an exponential relationship: T_grok proportional to exp(alpha rho) [1]. A single exponent, alpha near 7.5, described the delay across four different moduli with an R^2 of 0.996 [2]. Across the swept parameter ranges, manipulating the held norm shifted the grokking delay by about 19x, while changing the learning rate moved it by only about 2x [1]. Holding the norm above Wc slowed grokking but did not prevent it [2]. The exponential delay law vanished when a final LayerNorm was added to the architecture, which decouples the weight scale from the network's function. Removing the LayerNorm caused the exponential dependence to return [1]. The authors describe this pinned-norm delay as the exponential counterpart to the logarithmic delay predicted when the norm is allowed to contract freely [2]. arXiv, which hosts the paper, is an open-access repository of electronic preprints that has been operating since August 1991 and now receives about 24,000 submissions per month [6]. The arXivLabs framework, launched in 2020, allows community collaborators to build experimental tools that appear on article pages, such as citation explorers and code finders [4][5].