Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

Researchers have proposed two new frameworks to improve Reinforcement Learning from Human Feedback (RLHF), a technique used to align large language models with human preferences.

The first framework, FiMi-RM, aims to mitigate length bias in reward models used in RLHF. According to a paper submitted to arXiv on 19 May 2025^[1], RLHF often suffers from reward hacking, where policy learning exploits flaws in the trained reward model. FiMi-RM autonomously learns and corrects underlying bias patterns by deploying a lightweight fitting model to capture the non-linear relation between length and reward. Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution. The second framework, LCA, addresses the credit assignment challenge in outcome-supervised Process Reward Models (PRMs). A paper submitted to arXiv on 26 Jun 2026^[2] states that LCA formalizes outcome-supervised PRM as a Multiple Instance Learning (MIL) problem and introduces Softmax-Weighted-Sum (SWS) pooling. LCA is proven to be Bayes consistent under mild assumptions and consistently outperforms state-of-the-art outcome-supervised PRMs across multiple tasks and backbones.