Hide to Guide: Learning via Semantic Masking

A new training method called Semantic Masked Expert Policy Optimization (SMEPO) aims to curb reward hacking in reinforcement learning with verifiable rewards (RLVR) by selectively hiding reward-relevant content in expert demonstrations, according to research published on arXiv [1]. The technique, detailed in a paper from the MIT HAN Lab, addresses a persistent weakness in RLVR: when models are guided by expert traces, they can learn to copy the final answer or intermediate values directly rather than mastering the underlying reasoning [1]. This reward hacking channel undermines the purpose of using expert guidance in the first place [2]. Existing methods reduce this risk by truncating expert trajectories, but they control how much information is shown rather than precisely which parts should be hidden [2]. SMEPO instead applies a fine-grained semantic masking strategy that obscures reward-relevant spans along the critical path while preserving the expert’s decomposition, plan, and procedural structure [1]. The approach transforms difficult problems into a fill-in-the-blank exercise: the model can follow the expert’s problem-solving route but must reconstruct missing values, code, or entities on its own [2]. The method requires no changes to the reward function or the underlying RL objective [1]. Across math, code, and agentic search domains, SMEPO improved accuracy by up to 3.2 points over GRPO and reduced training time by up to 4.2x [1][2]. The concept of masking specific semantic spans draws a loose parallel to scope in computer programming, where the visibility of a name binding is deliberately constrained to a specific portion of source code to prevent unintended interactions [5]. The researchers have released the code publicly on GitHub [1].