Gefen: Optimized Stochastic Optimizer

A new optimizer called Gefen reduces the memory footprint of the widely used AdamW algorithm by roughly eight times without sacrificing performance, according to a paper posted to arXiv on June 11, 2026 [1]. The reduction corresponds to 6.5 GiB saved per billion model parameters [1]. AdamW is a default optimizer for modern deep learning, but its first- and second-moment states add approximately two parameter-sized buffers to training memory [1]. Gefen addresses that overhead through two mechanisms. It automatically shares second-moment estimates across parameter blocks, and it quantizes the first moment using a learned codebook [1]. The approach is motivated by a theoretical result showing that large mixed Hessian entries constrain the ratio of squared gradients toward one, which suggests that Hessian-aligned parameters are natural candidates for sharing second-moment statistics [2]. Because computing the Hessian directly at scale is impractical, Gefen infers block structure from the initial squared gradients, requiring no architecture-specific metadata or hyperparameters beyond AdamW defaults [3]. The algorithm learns an exact histogram-based dynamic-programming quantization codebook and reuses the same blocks for first-moment scaling, avoiding the introduction of an additional scaling constant [4]. Across diverse experiments, Gefen achieved the lowest peak optimizer memory among the compared AdamW-like methods while maintaining AdamW-level performance [1]. In FSDP and DDP training configurations, the reduced memory footprint enabled larger microbatches and improved throughput significantly over AdamW [5]. The authors describe Gefen as a practical drop-in replacement with lower memory usage that can increase throughput and enable training larger models or using larger batch sizes [1]. A complete Python implementation, including fused CUDA kernels, is available on GitHub [1]. The paper appeared on arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month and is not peer-reviewed [9].