NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

A new training-time method called NormGuard can prevent a subtle form of quality degradation that afflicts flow-based generative models after reinforcement learning fine-tuning, according to a preprint posted to arXiv. The technique targets “norm inflation,” a structural drift where the model’s internal velocity magnitudes creep upward during post-training. The preprint, submitted on 26 June 2026, identifies a consistent signature of perceptual quality loss in flow-matching generators that undergo reinforcement learning (RL) post-training [1]. Across three distinct post-training methods — NFT, AWM, and DPO — the per-step velocity norm increased by 5% to 15% relative to the reference model [2]. This inflation correlates with image artifacts that reward proxies fail to capture [2]. A related phenomenon has been documented in classifier-free guidance (CFG), where rescaling the velocity to a reference norm at inference time can mitigate artifacts [2]. The authors tested whether the same inference-time correction would work for RL-tuned models. It did not. Rescaling the velocity to match the reference norm at inference time neither improved reward nor fixed the quality degradation, because the inflation becomes co-adapted into the model weights during training [2]. An adjoint sensitivity analysis reinforced the finding: velocity magnitude rescaling carries no coherent first-order reward signal at the batch level [2]. Suppressing norm inflation is therefore unlikely to remove a component that consistently contributes to reward. Since inference-time renormalization fails and norm suppression carries no reward cost, the authors argue that training-time intervention is the appropriate strategy [2]. NormGuard implements this strategy as a hinge penalty that activates only when the velocity norm exceeds the reference norm. The penalty composes additively with any velocity-local base loss [2]. The method was evaluated across two base models, three post-training methods, and two reward proxies [2]. In all configurations, NormGuard consistently improved image quality and forensic realism as judged by multi-modal large language models (MLLMs), while preserving reward alignment [2]. The quality gains amplified under few-step inference and could not be explained by early stopping [2]. The work appears on arXiv, the open-access e-print repository that has hosted over two million articles since its founding in 1991 and currently receives about 24,000 submissions per month [6]. The preprint has not yet been peer-reviewed [6].