Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

Reinforcement learning causes less catastrophic forgetting than supervised fine-tuning when adapting language models to new tasks, according to a systematic comparison posted on arXiv. The study, by Howard Chen and collaborators, identifies the use of on-policy data as the key mechanism behind this difference [1][2]. The paper, titled "Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting," was initially submitted on 21 Oct 2025 and last revised on 26 Jun 2026 [1]. The research addresses a classic problem in machine learning where models degrade existing capabilities after post-training adaptation, a phenomenon known as catastrophic forgetting [1][2]. The authors tested two widely used post-training methods across different model families, including Llama and Qwen, and across tasks such as instruction following, general knowledge, and arithmetic reasoning [1][2]. Their experiments showed a consistent trend: reinforcement learning, or RL, led to less forgetting than supervised fine-tuning, or SFT, while achieving comparable or higher performance on the target task [1][2]. To understand why, the researchers modeled the language model as a mixture of two distributions—one for prior knowledge and one for the target task. They found that the mode-seeking nature of RL, which stems from its reliance on on-policy data, allows the model to keep prior knowledge intact while learning the new task [1][2]. The team then verified that the use of on-policy data, rather than other algorithmic choices like KL regularization or advantage estimation, underlies RL's robustness to forgetting in practical settings [1][2]. The findings also carry a practical implication. The study highlights that approximately on-policy data can be used to mitigate forgetting, and this data can be substantially more efficient to obtain than fully on-policy data [1][2]. The work was posted on arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month and hosts over two million articles [10]. The paper's abstract page also features experimental tools developed through arXivLabs, a framework launched in 2020 that enables community collaborators to build features such as citation explorers and recommender systems directly on the site [8][9].