Understanding helpfulness and harmless tension in reward models

Researchers have identified a neural mechanism underlying the tension between helpfulness and harmlessness in AI reward models, finding that shared neurons contribute to performance trade-offs when models are trained to satisfy both objectives simultaneously. The study, submitted to the arXiv preprint server on 11 Jun 2026, examines reward models used in reinforcement learning from human feedback (RLHF), a technique for aligning large language models with human preferences [1]. Reward models trained under mixed-objective settings often underperform those trained for a single objective, indicating interference between helpfulness and harmlessness [1]. Using activation-based methods, the researchers pinpointed neurons associated with each objective and tested their roles through targeted ablations [1]. The analysis showed that neurons supporting one objective frequently harmed the opposing one [1]. A substantial proportion of neurons were found to be shared between helpfulness and harmlessness, and these shared neurons exerted a disproportionate influence on model behavior, directly contributing to what the authors term alignment tension [1]. The findings arrive as the field grapples with the challenge of multi-objective alignment. Large language models, typically based on transformer architectures, are pre-trained to predict text and then fine-tuned to follow instructions and act as assistants [11]. Biased or inaccurate training data can make outputs less reliable, and benchmark evaluations attempt to measure model reasoning, factual accuracy, alignment, and safety [11]. arXiv, where the paper was posted, is an open-access repository of electronic preprints that are moderated but not peer-reviewed [9]. It hosts scientific papers across disciplines including computer science, physics, and mathematics, and as of November 2024 received about 24,000 submissions per month [9]. The platform also supports community-developed tools through its arXivLabs framework, which allows collaborators to build features such as citation explorers and code finders on article pages [8]. The study’s authors argue that understanding how alignment objectives are represented inside reward models is essential for building more controllable systems [1]. They suggest that future work should focus on disentangled alignment methods that can separate helpfulness and harmlessness signals, potentially reducing the interference observed in current mixed-objective training regimes [1].