PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration

A new post-hoc estimator called PEBS reduces reward-model calibration error by fitting per-annotator affine corrections, according to a preprint posted to arXiv on June 25, 2026 [1]. The method applies empirical-Bayes shrinkage to account for systematic differences in how human raters use rating scales, without retraining the underlying reward model [2]. Reinforcement Learning from Human Feedback, or RLHF, typically pools preference data from thousands of annotators and fits a single global calibrator. That average-rater fit can miss individual annotators who apply the rating scale with different offsets and slopes [2]. PEBS — short for Per-rater Empirical-Bayes Shrinkage — addresses the mismatch by fitting a separate affine calibrator for each rater on a held-out slice of their ratings, then shrinking those per-rater estimates toward the population mean using the Morris-James-Stein procedure [2]. The calculation is closed-form and does not require retraining the reward model; only the rater-level map used at inference time changes [2]. On the PRISM dataset, PEBS delivered an 8.58% reduction in within-user held-out root-mean-square error compared with the pooled population-slope baseline [1][2]. The result replicated on PluriHarms harm ratings, where the method posted a 9.66% RMSE reduction over the same baseline, using a Qwen-2.5 base model in-family [1][2]. Both figures are reported in the preprint abstract and confirmed in the full text [2]. The paper appeared on arXiv, the open-access e-print repository that hosts preprints across physics, mathematics, computer science, and related fields [6]. arXiv was founded in 1991 and now receives roughly 24,000 submissions per month [6]. The platform also supports community-built tools through its arXivLabs framework, which allows third-party developers to create features such as citation explorers and code linkers that appear on article abstract pages [5][4]. arXivLabs collaborators must adhere to the repository’s values of openness, community, excellence, and user-data privacy, and they receive only minimal, anonymized data necessary for their tools to function [5]. The PEBS preprint itself is indexed with several Labs integrations, including Bibliographic Explorer, Connected Papers, and the CORE Recommender, which help readers navigate related literature and discover relevant open-access research [4]. Those tools are separate from the paper’s technical contribution but reflect the broader infrastructure that supports dissemination and discussion of machine-learning research on the platform.