Robust and Fast Training via Per-Sample Clipping
- lab arXiv
- lab arXivLabs
- person Davide Nobile
A new gradient estimator, per-sample clipped SGD (PS-Clip-SGD), achieves optimal convergence rates for non-convex optimization problems under heavy-tailed gradient noise, according to a paper submitted to arXiv on May 4, 2026[1].
The PS-Clip-SGD method is based on per-sample gradient clipping and has been shown to outperform vanilla SGD with momentum and standard gradient clipping when training AlexNet on the CIFAR-100 dataset. The authors of the paper, available on arXiv[1], demonstrate that per-sample clipping can improve training performance when applied at the mini-batch level, incurring virtually no additional computational cost. In a related development, researchers have also made advancements in linear TD(0) algorithms, showing that a single stepsize schedule can achieve both robust and fast convergence rates. This is accomplished using Polyak-Ruppert averaging with a stepsize schedule proportional to 1 / (τ_mix * log(t) * sqrt(t)), achieving a robust curvature-free rate of O(τ_mix / sqrt(T)) and a fast curvature-dependent rate of O(τ_mix^2 / ωT)[2].
research-paperapplicationinfrastructure
Background sources we checked (10)
- arxiv.org ↗ We propose a robust gradient estimator based on per-sample gradient clipping and analyze its properties both theoretically and empirically. We show that the resulting method, per-sample clipped SGD (PS-Clip-SGD), achieves optimal in-expectation convergence rates for non-convex op…
- en.wikipedia.org ↗ Path tracing is a rendering algorithm in computer graphics that simulates how light interacts with objects and participating media to generate realistic (physically plausible) images. It is based on earlier, more limited, ray tracing algorithms. Path tracing is used to create pho…
- en.wikipedia.org ↗ In telecommunications, orthogonal frequency-division multiplexing (OFDM) is a type of digital transmission used in digital modulation for encoding digital (binary) data on multiple carrier frequencies. OFDM has developed into a popular scheme for wideband digital communication, u…
- en.wikipedia.org ↗ Rendering is the process of generating an image from input data such as 3D models. The word "rendering" (in one of its senses) originally meant the task performed by an artist when depicting a real or imaginary thing (the finished artwork is also called a "rendering"). Today, to …
- en.wikipedia.org ↗ Temporal envelope (ENV) and temporal fine structure (TFS) are changes in the amplitude and frequency of sound perceived by humans over time. These temporal changes are responsible for several aspects of auditory perception, including loudness, pitch and timbre perception and spat…
- info.arxiv.org ↗ arXiv Labs - arXiv info | arXiv e-print repository Skip to content # arXiv Labs Attention arXiv Users: arXiv Labs is pausing new proposals ## What are arXiv Labs? arXiv Labs are a way for the community to contribute new, useful features to arXiv. These integrations are avail…
- blog.arxiv.org ↗ arXivLabs: a space for community innovation – arXiv blog arXiv has launched a new, formalized framework enabling innovative collaborations with individuals and organizations. “Members of our community want to contribute tools that enhance the arXiv experience, and we val…
- info.arxiv.org ↗ arXivLabs: Showcase - arXiv info | arXiv e-print repository ... # arXivLabs: Showcase ... arXiv is surrounded by a community of researchers and developers working at the cutting edge of information science and technology. ... While the arXiv team is focused on our core mission—pr…
- en.wikipedia.org ↗ arXiv (pronounced as "archive"—the X represents the Greek letter chi ⟨χ⟩) is an open-access repository of electronic preprints and postprints (known as e-prints) approved for posting after moderation, but not peer reviewed. It consists of scientific papers in the fields of mathem…
- en.wikipedia.org ↗ 14 (fourteen) is the natural number following 13 and preceding 15.…