Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output

A new method called Graph-based Advantage Estimation, or GraphAE, taps into the hidden states of reward models to improve reinforcement learning from human feedback, according to a preprint posted to arXiv on June 9 [1]. The technique moves beyond noisy scalar rewards by using richer semantic signals already present inside the model. Standard reinforcement learning from human feedback pipelines train a reward model and then use its scalar output to guide policy updates. Those scalar rewards are often noisy and fail to capture fine-grained preference differences [1]. The authors argue that the reward model’s hidden states encode far richer semantic and preference information that goes unused when only the final scalar is extracted [2]. GraphAE treats each sampled group of responses as a graph. Nodes correspond to individual responses, and edge weights reflect pairwise similarity measured in the reward model’s hidden space. Advantages are then computed through graph propagation, allowing each sample to incorporate contextual information from its neighbors [3]. The method solves a graph-regularized objective that balances fidelity to the original scalar rewards with smoothness over the graph, yielding a closed-form solution that can be efficiently solved in practice [4]. The estimator is lightweight and can be dropped into existing group-based reinforcement learning algorithms by replacing the original advantage estimator. The authors integrated GraphAE into GRPO, GSPO, and RLOO and ran experiments across multiple models and benchmarks [1]. On Arena-Hard-v0.1 the method posted a gain of up to +6.3, on AlpacaEval 2.0 a gain of up to +8.27, and on MT-Bench a gain of up to +0.22 [1]. Other recent work has also sought to move beyond scalar rewards. One framework, Reinforcement Learning with Relative Rewards, shifts reward shaping from absolute valuation toward relative ordering by integrating intra-group ranking information directly into advantage computation [5]. Another approach, General Preference Reinforcement Learning, treats a generalized preference model as a structured, multi-dimensional reward source and computes per-dimension group-relative advantages across subspaces, normalizing each on its own scale before aggregation [7]. GraphAE differs by keeping the reward model intact and extracting additional signal from its internal representations rather than redesigning the reward structure. The preprint was posted on arXiv, an open-access repository that hosts e-prints across physics, mathematics, computer science, and related fields and receives about 24,000 submissions per month as of late 2024 [11]. The authors have released code for GraphAE on GitHub [3].