DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning

A new training framework called DRA-GRPO aims to improve mathematical reasoning in large language models by encouraging diverse solution strategies, according to a paper posted on arXiv. The method adjusts the reward signal during post-training to prevent models from collapsing onto a narrow set of reasoning patterns [1]. The framework, Diversity-aware Reward Adjustment for Group Relative Policy Optimization, addresses a limitation in standard GRPO, a reinforcement learning technique used to enhance mathematical reasoning in LLMs after initial training [1]. Standard GRPO assigns scalar correctness rewards that can be identical for semantically distinct reasoning paths, a problem the authors term Diversity-Quality Inconsistency. This causes the model's policy to favor a limited set of dominant strategies while discarding equally valid but structurally novel approaches [1]. DRA-GRPO calibrates the reward signal by measuring the semantic density of sampled reasoning groups using Submodular Mutual Information. It implements an Inverse Propensity Scoring mechanism that the authors describe as creating a "repulsive force against redundancy" to drive broader coverage of high-reward strategies [1]. The method is designed to integrate with existing GRPO variants without architectural changes [1]. In empirical tests across five math benchmarks, DRA-GRPO achieved an average accuracy of 58.2% on DeepSeek-R1-Distill-Qwen-1.5B, a distilled version of a model from the Chinese AI company DeepSeek [1][7]. The training used only 7,000 samples and cost $55, which the authors present as evidence of data-efficient alignment [1]. DeepSeek, founded in 2023 and based in Hangzhou, has drawn industry attention for producing competitive models at reported costs far below those of larger rivals [7]. The Qwen model family used in the experiments is developed by Alibaba Cloud and distributed under open-source licenses [9]. The paper, authored by Xiwen Chen and colleagues, has been revised several times since its initial submission in May 2025, with the latest version posted in June 2026 [1]. The code repository is publicly available on GitHub [1]. The work contributes to a growing body of research on post-training optimization for LLMs, a category of machine learning models with many parameters trained on vast text corpora for language generation tasks [8].