Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization

A team of researchers has proposed a new reinforcement-learning method, Intrinsic Signal Policy Optimization (ISPO), designed to fix two structural failure modes that limit how large language models learn long-chain reasoning from verifiable rewards [1]. The method targets weaknesses in Group Relative Policy Optimization (GRPO), a widely used approach in reinforcement learning with verifiable rewards (RLVR). GRPO relies on a binary outcome signal, which the authors say leads to two problems: Zero-Advantage Collapse, where all candidate outputs in a group receive the same reward and the training gradient vanishes, and Hallucinated Certainty, where a model grows increasingly confident in incorrect answers late in training [2]. ISPO densifies the reward by adding intrinsic signals derived entirely from the policy’s own conditional probabilities. It combines a sequence-level signal that measures how informative a reasoning trajectory is for the final answer with a token-level directional reward that penalizes confidently wrong predictions at critical decision tokens [2]. In experiments across three base models and five mathematical reasoning benchmarks, ISPO outperformed competitive baselines. The largest performance gains appeared on the hardest benchmarks, where Zero-Advantage Collapse is most frequent, and diagnostics of training dynamics confirmed that both failure modes were reduced [2]. The work was submitted to arXiv in June 2026 [1]. RLVR has become a prominent paradigm for eliciting reasoning in large language models, a class of machine-learning systems with many parameters trained on vast text corpora [8]. The field has seen rapid commercial and open-source development. Chinese firm DeepSeek, founded in 2023, drew attention in January 2025 when its DeepSeek-R1 model delivered responses comparable to OpenAI’s GPT-4 and o1 while reportedly using far less computing power [7]. Alibaba Cloud’s Qwen family of models, many released under the Apache 2.0 license, represents another open-weight effort in the same space [9]. Research on verifier-in-the-loop reinforcement learning extends beyond text. A 2026 scoping review of generative systems for quantum circuit and code generation organized the field along artifact type and training regime, including verifier-in-the-loop RL, and found that no system reported end-to-end evaluation on quantum hardware [3]. The ISPO paper appears on arXiv, a preprint server that since 2022 has integrated with Hugging Face Spaces to let authors and the community attach interactive demos to papers, increasing reproducibility and public engagement [5][6].