A Unifying Lens on Reward Uncertainty in RLHF

Researchers have introduced new approaches to reinforcement learning from human feedback (RLHF) and supervised fine-tuning, addressing issues such as reward hacking and uncertainty in reward models.

Two papers submitted in 2026 propose novel methods to improve RLHF and supervised fine-tuning. The first paper addresses reward hacking by using a distributional reward model and pessimism^[1]. RLHF is currently bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. The proposed distributional reward model provides a principled notion of uncertainty, unifying prior heuristics for RM ensemble aggregation, including mean aggregation, worst-case optimization, and uncertainty-weighted optimization. The second paper introduces a new framework, Q-target, which reinterprets supervised fine-tuning as target distribution design^[2]. Q-target decomposes SFT supervision into two explicit choices: how strongly to rely on the observed token, and how to allocate the remaining probability mass over alternatives. The Q-target framework was evaluated across ten dataset-model settings and outperformed existing variants. Supervised fine-tuning typically maximizes the likelihood of every token in a demonstrated trajectory, but an observed token can be non-unique, noisy, or misaligned with the model prior.