Deep Neural Network Training as Random Effects: An Optimization-Inference Duality

A new statistical framework recasts the training of deep neural networks as a classical random-effects inference problem, showing that the gradient-flow path under the neural tangent kernel is exactly equivalent to an empirical Bayes inference trajectory, according to a paper submitted to arXiv on 27 May 2026 [1]. The work addresses a long-standing gap in the theoretical understanding of deep neural networks (DNNs). While DNNs have achieved what the authors describe as "remarkable empirical success," their training dynamics have been analyzed predominantly through the lens of optimization rather than statistical principles [1]. The neural tangent kernel (NTK), introduced in 2018 by Jacot, Gabriel, and Hongler, already established that in the limit of infinite layer width, training a neural network with gradient descent becomes equivalent to kernel regression with the NTK [3]. The new framework builds on this duality by demonstrating that the continuous-time NTK gradient flow produces predictions exactly matching those from a random-effects model [1]. Under this formulation, training time is reinterpreted as a variance component—or an empirical Bayes covariance hyperparameter—that governs how variation is allocated between noise and structured signal [1]. The gradient-flow path therefore serves simultaneously as an optimization trajectory and as an empirical Bayes inference path. Conditional on a given training time, the network's output corresponds to the posterior mean of the latent signal [1]. The framework yields a two-stage inferential procedure. First, a variance-component test determines whether DNN training captures statistically significant structure beyond random initialization. If training is warranted, the second stage uses restricted maximum likelihood (REML) to estimate the optimal training time, transforming early stopping from an externally tuned heuristic into a likelihood-based empirical Bayes decision [1]. The REML-derived stopping time has a spectral interpretation in the NTK eigenbasis: training continues until spectral loss decorrelation is achieved [1]. The authors further establish that REML-guided early stopping achieves asymptotically optimal prediction error for fixed-design in-sample prediction and, under additional regularity conditions, for out-of-sample prediction [1]. The paper provides a principled statistical foundation for deciding both whether and how long to train deep neural networks, a question of practical importance as models scale to the size of modern large language models, which are themselves neural networks trained on vast text corpora [5].