Robust and Fast Training via Per-Sample Clipping

A new optimization method called per-sample clipped SGD achieves optimal convergence rates for non-convex problems under heavy-tailed gradient noise, according to a paper posted to the arXiv preprint server [1]. The paper, authored by Davide Nobile and submitted on 4 May 2026, introduces a robust gradient estimator based on per-sample gradient clipping [1]. The resulting algorithm, PS-Clip-SGD, is shown to attain optimal in-expectation convergence rates for non-convex optimization problems when gradient noise follows a heavy-tailed distribution [2]. The authors also establish high-probability convergence guarantees that match the in-expectation rates up to polylogarithmic factors in the failure probability [2]. The theoretical findings are supported by numerical experiments. When training the AlexNet architecture on the CIFAR-100 dataset, PS-Clip-SGD outperformed both vanilla SGD with momentum and standard gradient clipping, even after accounting for the additional computational time required by per-sample clipping [2]. The work also examines the interaction between per-sample clipping and gradient accumulation. The researchers found that applying clipping at the mini-batch level, rather than waiting until all accumulation steps are complete, can improve training performance while incurring virtually no additional computational cost [2]. This observation runs counter to common practice [2]. The paper appeared on arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month [10]. The repository, founded in 1991, hosts papers across mathematics, physics, computer science, and related fields, and is not peer-reviewed [10]. The manuscript, which spans 362 KB, was revised on 23 June 2026 [1]. Per-sample clipping has drawn attention in optimization research because heavy-tailed gradient noise can destabilize standard stochastic gradient descent. By clipping each sample's gradient individually before averaging, PS-Clip-SGD limits the influence of extreme gradient values. The paper's convergence guarantees place the method among the theoretically strongest approaches for this problem class [2].