Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift

A team of researchers has proposed ReGrad, a method that treats gradients as retrievable units of knowledge to enable continual post-training of models without accumulating weight drift, according to a paper submitted in 2026 [1]. The approach, detailed on the preprint server arXiv, pre-computes document-specific gradients offline and stores them in an indexed Gradient Bank. At inference time, only query-relevant gradients are retrieved for temporary weight adaptation [1]. The paper states that ReGrad outperforms both continual post-training (CPT) and retrieval-augmented generation (RAG) baselines in experiments across general and domain-specific settings [1]. Continual post-training allows models to absorb emerging knowledge after deployment, but repeatedly updating shared parameters can cause catastrophic forgetting and degrade general capabilities [2]. Retrieval-augmented generation avoids such parameter drift, yet the authors note it often lacks the depth of parametric knowledge integration [2]. ReGrad aims to combine the benefits of both approaches by enabling scalable and reversible parametric knowledge injection without accumulating weight drift [2]. A key technical challenge addressed in the work is that raw language-modeling gradients are optimized for token-level document reconstruction rather than for query-driven knowledge use. To resolve this, the researchers introduce a bi-level meta-learning objective that reshapes document-derived gradients into generalizable adaptation signals for downstream tasks [2]. Neural networks, which underpin modern large language models, consist of connected units called artificial neurons. The strength of the signal at each connection is determined by a weight, which adjusts during training [3]. Large language models are trained with self-supervised learning on vast amounts of text and contain many parameters [9]. The ReGrad method intervenes at the gradient level, treating these weight-update signals as discrete, storable knowledge units rather than applying them permanently to the model [1]. The paper was submitted to arXiv on 14 June 2026 [1]. arXiv, an open-access repository of electronic preprints, was founded in 1991 and surpassed two million articles by the end of 2021 [7]. The repository allows researchers to share findings before peer review, with a submission rate of about 24,000 articles per month as of November 2024 [7].