PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

A new token-level reinforcement learning framework called Perception-Reinforced Policy Optimization (PRPO) has been proposed to improve the reasoning of Large Vision-Language Models by targeting the weak supervision of visually grounded tokens, according to a paper published on arXiv [1]. The framework addresses a limitation in current Reinforcement Learning with Verifiable Rewards (RLVR) methods, which typically assign identical learning signals across all generated tokens in a reasoning trajectory [1]. The researchers argue this coarse-grained credit assignment is mismatched to multimodal reasoning, where only a sparse subset of tokens is causally grounded in visual evidence [1]. As a result, pivotal perceptual tokens receive weak supervision and are often overwhelmed by language priors or reasoning-template tokens [1]. To counter this, PRPO introduces two core components: Robust Visual Dependency (RVD) and Perceptual Advantage Reshaping (PAR) [1]. RVD is a metric designed to identify tokens whose predictions are both visually grounded and stable under perturbation, filtering out brittle or noisy visual tokens [1]. PAR then uses this identification to amplify learning signals for perceptually informative tokens while preserving stable gradients for non-perceptual tokens [1]. The approach was tested across seven multimodal reasoning benchmarks [1]. The paper reports that PRPO consistently outperformed strong LVLM baselines at both the 3B and 7B model scales, achieving average gains of 23.3% and 21.1%, respectively [1]. The authors state the method achieves state-of-the-art performance with improved training efficiency and stronger cross-task generalization [1]. The findings underscore the importance of fine-grained credit assignment for scalable multimodal reinforcement learning, moving beyond the trajectory-level outcome rewards that have defined prior work in the field [1].