Explicit Critic Guidance for Aligning Diffusion Models

A new framework for aligning diffusion models with non-differentiable objectives uses a state-aligned latent actor-critic design that enables trajectory-level PPO training and stable value-based optimization, according to research submitted in 2026 [1]. The method, described in a paper posted to arXiv, allows the diffusion model to function as its own timestep-conditioned value function, predicting values directly on noisy latent states [2]. This architecture supports stable actor-critic optimization through simple conditioning and value pretraining strategies, and the learned critic can be reused for inference-time steering [2]. The work extends to multi-reward optimization, where joint training with complementary rewards helps mitigate reward hacking [2]. Across both UNet- and DiT-based backbones, the approach consistently outperformed prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks, with test-time steering providing additional gains in generation quality [2]. Online reinforcement learning has become increasingly important for aligning generative models with objectives that cannot be differentiated, a challenge that has grown alongside the wider deployment of generative AI systems [4]. Since the 2020s, generative AI has become widely available to produce images, audio, and video from text prompts, and major labs now pursue artificial general intelligence capable of completing virtually any cognitive task at least as well as a human [4]. The new actor-critic framework addresses a persistent limitation in existing methods: the difficulty of assigning fine-grained credit along denoising trajectories [2]. The research was submitted through arXivLabs, a framework that allows collaborators to develop and share new arXiv features directly on the platform [1]. The paper's abstract notes that the method "enables trajectory-level PPO training" and "naturally allows the learned critic to be reused for inference-time steering" [2]. The multi-reward extension is designed to alleviate reward hacking, a known failure mode in reinforcement learning where agents exploit reward functions in unintended ways [2]. The broader context for such alignment research includes ongoing discussions about AI safety, unintended consequences, and potential existential risks, which have prompted calls for AI regulation [4]. As generative models become more capable, techniques that align their outputs with human preferences and non-differentiable quality metrics have drawn increasing attention from both academic and industry researchers [4].