Beyond a Single Explanation of the Adam--SGD Gap
- lab arXiv
- lab arXivLabs
- person Sam Altman
Two recent studies on arXiv examine the performance gaps between Adam and SGD optimization algorithms and the effectiveness of INLP in safety fine-tuned chat models.
A controlled empirical study across various tasks and architectures found that no single factor consistently explains the performance gap between Adam and SGD[1]. The Adam advantage can persist or disappear depending on vocabulary distribution, and even reverse in favor of SGD in softmax-attention models. The study, submitted on 12 Jun 2026, also observed that a crossover batch size exists where the relative advantage shifts from SGD to Adam as batch size scales. Another study published on arXiv found that INLP can match Diff-in-Means (DiM) at steering refusal in safety fine-tuned chat models, and that refusal is mediated by a single linear direction in the residual stream[2]. INLP counterfactual flipping was competitive with DiM directional ablation on refusal suppression, while nullspace projection was consistently weaker. Restricting INLP to leading directions preserved most of the suppression effect at near-baseline perplexity.
research-papercommentary
Background sources we checked (8)
- arxiv.org ↗ Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties. Yet these explanations are often studied in isolation, leaving their relative importance unclear. In…
- en.wikipedia.org ↗ Dilophosaurus ( dy-LOH-fə-SOR-əs, -foh-) is a genus of theropod dinosaurs that lived in what is now North America during the Early Jurassic, about 186 million years ago. Three skeletons were discovered in northern Arizona in 1940, and the two best preserved were collected in 194…
- info.arxiv.org ↗ arXiv Labs - arXiv info | arXiv e-print repository Skip to content # arXiv Labs Attention arXiv Users: arXiv Labs is pausing new proposals ## What are arXiv Labs? arXiv Labs are a way for the community to contribute new, useful features to arXiv. These integrations are avail…
- blog.arxiv.org ↗ arXivLabs: a space for community innovation – arXiv blog arXiv has launched a new, formalized framework enabling innovative collaborations with individuals and organizations. “Members of our community want to contribute tools that enhance the arXiv experience, and we val…
- info.arxiv.org ↗ arXivLabs: Showcase - arXiv info | arXiv e-print repository ... # arXivLabs: Showcase ... arXiv is surrounded by a community of researchers and developers working at the cutting edge of information science and technology. ... While the arXiv team is focused on our core mission—pr…
- en.wikipedia.org ↗ arXiv (pronounced as "archive"—the X represents the Greek letter chi ⟨χ⟩) is an open-access repository of electronic preprints and postprints (known as e-prints) approved for posting after moderation, but not peer reviewed. It consists of scientific papers in the fields of mathem…
- en.wikipedia.org ↗ 14 (fourteen) is the natural number following 13 and preceding 15.…
- en.wikipedia.org ↗ A large language model (LLM) is a type of machine learning model designed for natural language processing tasks such as language generation. LLMs are language models with many parameters, and are trained with self-supervised learning on a vast amount of text.…