Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy

A team of researchers has reformulated reinforcement learning for large language models to address a persistent train-inference mismatch that can cause training collapses and sub-optimal performance, according to a paper submitted to arXiv in June 2026 [1]. The work targets a hidden discrepancy between the engines and architectures used during training and those used during inference, which recent findings have linked to unpredictable failures in reinforcement learning (RL) post-training pipelines [1][2]. The authors report that a training policy can self-correct this mismatch when given an appropriate learning signal [2]. They further identify a discrepancy tolerance region: aggressively narrowing the discrepancy inside this zone can suppress policy exploration and reduce learning efficiency, while reducing excessive discrepancy outside it improves optimization consistency and raises the achievable local performance ceiling [2]. To operationalize these insights, the researchers formulate the problem as a Discrepancy-Constrained Markov Decision Process, or DCMDP, which couples reward maximization with a constraint that aligns training and inference behavior [1][2]. A Lagrangian relaxation mechanism dynamically adjusts the relative weight of performance improvement and discrepancy control based on the current degree of violation, allowing the policy to explore freely within the tolerance region while guiding it back when the discrepancy exceeds a safe boundary [2]. The approach was tested on an 8-billion-parameter dense model, Qwen-3-8b, and a 30-billion-parameter Mixture-of-Expert model, Qwen-3-30bA3b, with the authors reporting significant performance improvements for both [1][2]. The framework also enables a heterogeneous training paradigm in which large language models can be optimized in high-fidelity training setups while being explicitly aligned for low-cost, resource-constrained inference deployment [2]. Large language models are neural networks trained on vast text corpora for tasks such as generation, summarization, and translation, and they underpin modern chatbots [8]. The paper was posted on arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month and hosts more than two million articles [6]. Submissions to arXiv are moderated but not peer-reviewed [6].