The Representation-Rationalizability Tradeoff in Reward Learning

Researchers are exploring new methods in reward learning and reinforcement learning (RL) to overcome the significant bottleneck of manual reward design in applying RL to real-world problems.

Reward learning infers reward functions from human feedback rather than manually specified ones, according to a paper submitted to arXiv on 29 May 2026^[2]. The traditional approach to reward design is a major obstacle in applying RL to real-world problems, as noted in another recent arXiv submission^[1]. A new method, R4, offers formal guarantees, with a provably minimal and complete solution set under mild assumptions, and consistently matches or outperforms existing RL methods on robotic benchmarks^[1]. However, research also highlights challenges in reward learning, particularly with heterogeneous annotator samples, which can lead to pooled preferences with Condorcet cycles, making it impossible for a scalar reward to consistently evaluate all compared response pairs^[2]. The study further shows that the excess cross-entropy loss of any reward built on a learned representation decomposes into a representational term and an aggregation term, with a richer representation shrinking the former but enlarging the latter. This tradeoff is also observed in direct preference optimization (DPO), and jointly training the embedding and the reward does not guarantee finding the optimal balance.