OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

A new algorithm called Off-policy Generative Policy Optimization (OGPO) enables sample-efficient fine-tuning of generative control policies for robots, according to a preprint posted to the arXiv repository [1][2]. The method uses off-policy critic networks to reuse data and propagate policy gradients through the full generative process [2]. Generative control policies, including diffusion- and flow-based approaches, have become effective parameterizations for robot learning [2]. The OGPO algorithm, detailed in a paper submitted to arXiv on 4 May 2026 and last revised on 26 June 2026, maintains off-policy critic networks to maximize data reuse [1][2]. It propagates policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward [2]. The authors report state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control [2]. They state that OGPO is the only method they know of that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning [2]. The paper also demonstrates that OGPO outperforms alternative methods on policy steering and learning residual corrections [2]. To address critic over-exploitation, the researchers introduce several stabilization techniques: success-buffer regularization, two-sided conservative advantages, and Q-variance reduction [2]. These are applied across both state- and pixel-based settings [2]. Beyond proposing the algorithm, the work includes a systematic empirical study of generative control policy fine-tuning, identifying stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement [2]. The paper was posted on arXiv, an open-access repository of electronic preprints that is not peer-reviewed [6]. Founded in 1991, arXiv passed the two-million-article milestone by the end of 2021 and as of November 2024 receives about 24,000 submissions per month [6]. The repository hosts papers in fields including computer science, physics, and mathematics [6]. The submission history shows the paper was submitted by Sarvesh Bipin Patil, with the initial version weighing 42,420 KB and subsequent revisions reduced to roughly 21,000 KB [1].