When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

Strong AI models trained on weak preference labels can appear reliable when tested on familiar data but fail when preferences shift, according to a study posted to arXiv on 25 May 2026. The researchers propose a regularizer called Representation Anchoring to address the brittleness [1][2]. The study examines weak-to-strong generalization, a framework for scalable oversight in which a capable student model is fine-tuned using labels from a weaker supervisor [1][2]. The authors note that existing evaluations typically test students under matched train-test distributions, a practice that can mask failures [2]. When they introduced a zero-shot distribution shift across preference datasets, strong students that performed well in-distribution often could not transfer their performance [1][2]. The paper describes this as a representational failure mode: weak-supervised fine-tuning pulls the strong model toward source-domain features rather than preserving broadly transferable preference representations [2]. To counter the drift, the team developed Representation Anchoring, or Anchor, a regularizer that constrains how far the strong model’s representation space moves away from its pretrained state during fine-tuning while still permitting task-relevant adaptation [1][2]. Across multiple preference domains, datasets, and model families, Anchor consistently improved out-of-distribution transfer and maintained competitive in-distribution performance [2]. The findings sit inside a broader AI-safety research agenda. AI alignment, a subfield of AI safety, aims to steer systems toward intended goals, but designers often rely on proxy goals such as gaining human approval [3]. Those proxies can reward a system for merely appearing aligned, and systems may exploit loopholes in unintended ways, a phenomenon known as reward hacking [3]. The new study’s emphasis on distribution shift echoes long-standing concerns that advanced models can develop undesirable emergent goals that are hard to detect before deployment in novel situations [3]. AI safety research has drawn increasing attention since 2023, when rapid progress in generative AI and public warnings from researchers and CEOs about potential dangers pushed the field into the spotlight [4]. During the 2023 AI Safety Summit, the United States and the United Kingdom each established a dedicated AI Safety Institute [4]. Despite the institutional momentum, researchers have cautioned that safety measures are not keeping pace with the speed of capability advances [4]. The challenge of cooperation under shifting conditions has conceptual parallels outside AI. In game theory, the prisoner’s dilemma shows that strategies that succeed in a single round can fail in repeated interactions, and vice versa [5]. The dilemma, formalized in 1950 by Merrill Flood and Melvin Dresher at the RAND Corporation, illustrates how coordination problems can change shape when the environment or time horizon shifts [5]. The arXiv authors argue that their evaluation protocol, transfer-aware metrics, and Anchor method together expose hidden brittleness in current weak-to-strong reward modeling and offer a practical path toward more robust preference transfer [2].