Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance

A new statistical method constructs confidence regions from stochastic gradient descent trajectories without requiring prior knowledge of whether the data has finite or infinite variance, according to a paper posted to arXiv on 25 May 2026 [1]. Stochastic gradient descent, or SGD, is a foundational algorithm for large-scale statistical learning and stochastic optimization [1]. The algorithm iteratively updates model parameters by moving in the direction of the negative gradient estimated from small batches of data. However, quantifying the uncertainty of the resulting estimates has remained difficult, particularly when the stochastic gradients exhibit heavy-tailed behavior and infinite variance. In such regimes, the limiting distributions of the estimates depend on unknown nuisance parameters, making standard inference procedures unreliable [1]. The proposed methodology is model-agnostic and does not require explicit estimation of tail indices, slowly varying functions, or stable-law parameters [1]. The procedure rests on a joint weak convergence result for the Polyak-Ruppert averaged estimator and an empirical second-moment normalizer constructed from the stochastic gradients collected along the SGD trajectory. This joint limit produces a self-normalized statistic in which the leading tail-dependent scaling terms cancel, yielding a pivotal quantity whose distribution can be approximated without knowing the tail behavior of the gradients [1]. To obtain critical values, the authors employ a subsampling calibration scheme. Subsampling uses overlapping blocks of the observed SGD iterates to approximate the sampling distribution of the self-normalized statistic, bypassing the need to estimate tail indices or other heavy-tail parameters [1]. The resulting confidence regions are asymptotically valid under both finite and infinite second-moment regimes. Simulation studies reported in the paper show reliable coverage across a range of settings, supporting the method as a practical tool for uncertainty quantification in stochastic optimization [1]. The broader challenge of balancing bias and variance is central to statistical learning. The bias–variance tradeoff describes how increasing model flexibility can reduce training error but may increase the variability of the parameter estimates when applied to new data [3]. High variance can cause a model to fit random noise in the training set, a phenomenon known as overfitting, while high bias can lead to underfitting by missing relevant relationships between features and targets [3]. Reliable confidence regions, such as those proposed in the new methodology, offer a way to assess whether an estimate reflects a stable pattern or is an artifact of sampling variability. SGD-based optimization underpins many modern machine learning systems, including the training of diffusion models and variational autoencoders. Diffusion models learn to generate data by reversing a process that gradually adds Gaussian noise, and they are typically trained using variational inference [4]. Variational autoencoders, introduced in 2013, map inputs to distributions in a latent space rather than to single points, a design that helps avoid overfitting the training data [5]. In both cases, practitioners rely on SGD or its variants to optimize high-dimensional parameter spaces, and the new inference procedure could provide principled uncertainty estimates for the resulting models.