ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

Researchers have introduced ExTra, a framework designed to improve reinforcement learning for language models by extracting exploration signals from the model's own generated outputs, addressing a failure mode in current training methods [1]. The framework, detailed in a paper submitted to arXiv, targets a specific weakness in Reinforcement Learning with Verifiable Rewards (RLVR). This approach can stall when prompts are too easy, producing groups of all-correct but low-diversity rollouts with little gradient signal, or when prompts are too hard, yielding all-incorrect groups with no positive reward [1][2]. ExTra, which stands for Exploratory Trajectory Optimization, is compatible with the Group Relative Policy Optimization (GRPO) algorithm [1][2]. It combines two mechanisms to overcome these exploration failures. The first is a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions. The second is entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps [1][2]. The researchers tested ExTra on the Qwen3-1.7B model across six mathematical reasoning benchmarks. The results showed an improvement of about 5 points on pass@1 and 7 points on pass@16 compared to a standard GRPO baseline [1][2]. This demonstrates that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage [1][2]. The work suggests a path toward more robust training for reasoning models, where the system learns not just from correct answers but from the diversity of its own problem-solving attempts [1].