The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning

A new study identifies a "Quality-Utility Paradox" in AI training, finding that data refined by a more powerful model to achieve higher reward scores can actually impair the mathematical reasoning of smaller language models. The research, submitted in 2026, challenges a common assumption in knowledge distillation, a process where a powerful "Oracle" model is used to improve a Small Language Model (SLM) [1]. The standard practice assumes that training traces with higher reward model scores provide more useful supervision [2]. However, the paper demonstrates a counterintuitive outcome: data refined or synthesized by a stronger Oracle consistently underperforms traces generated by the SLM itself and selected through rejection sampling [2]. This effect was observed across the Qwen2.5, LLaMA-3, and DeepSeek model families [1]. DeepSeek, a Chinese AI company, is known for developing cost-effective, open-weight large language models that have rivaled those from larger U.S. firms [5]. The study's authors explain that Oracle refinement couples logical repair with a "distributional drift" away from the SLM's native reasoning distribution [2]. This drift increases the learner's adaptation cost, which can outweigh the benefit of improved reasoning logic [1]. The phenomenon echoes principles from behavioral economics, a field that studies how decisions deviate from those implied by traditional economic theory due to cognitive and psychological factors [3]. In this case, the AI system's learning process is not purely rational; the context and style of the information matter as much as its factual quality. To address this, the researchers introduced "Style-Aligned Refinement" [1]. This technique preserves the native reasoning trajectory of the SLM while retaining the logical repair from the Oracle [2]. The intervention was shown to lower the adaptation cost and restore downstream utility [1]. The findings suggest that effective mathematical reasoning distillation should jointly optimize perceived solution quality and learner-data compatibility, rather than relying solely on reward-model scores [2]. The datasets and code for the project have been made publicly available on GitHub [2].