How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis

A new theoretical analysis examines how neural reward models learn features when the learned reward is exponentiated to define a deployed policy, a feedback loop common in KL-regularized policy optimization [1]. The study, posted to arXiv on 23 May 2026, uses a Gaussian single-index model where the true reward is a function of a hidden direction and inputs are drawn from N(0, I_d) [1][2]. The authors propose a two-stage neural reward model: the first stage learns the hidden direction from reward-weighted samples, and the second stage fits the readout layer via weighted ridge regression [1][2]. Exponential reward weighting alters the Hermite signal available to the first layer. For any feature-learning temperature β1 above a dimension-free O(1) threshold, a constant fraction of neurons recover the hidden direction, with weak-recovery complexity governed by the generative exponent [1][2]. After feature recovery, the paper derives tilted-policy value-gap bounds for two weighting schemes: an idealized label-weighted fit with weights e^{y/β2} and a more practical surrogate-weighted fit with weights e^{r_{a_0}(x)/β2} [1][2]. Keeping the β2-dependence explicit yields an admissible set of deployment temperatures, balancing the gain from lowering β2 against the learning cost amplified by exponential weighting. In the surrogate-weighted case, proxy-dependent factors shrink this admissible set [1][2]. Reward modeling underpins many large language model alignment pipelines. Large language models, typically built on transformer architectures, are trained on vast text corpora and evaluated on reasoning, factual accuracy, and safety benchmarks [3]. The feedback loop studied in the paper — where a learned reward is exponentiated to shape a policy — mirrors challenges in recommender systems, which use machine learning to analyze user behavior and personalize content feeds across streaming, e-commerce, and social media platforms [5]. Decision-making research, which examines how beliefs and preferences drive choices, provides a broader cognitive framing for policy optimization under uncertainty [4]. The work was developed within arXivLabs, a framework that lets community collaborators build and share new features on the arXiv platform [1].