Steerable Cultural Preference Optimization of Reward Models

A new training algorithm called Steerable Cultural Preference Optimization (SCPO) aims to make large language models more responsive to diverse cultural subcommunities, according to research published on arXiv [1]. The method separately evaluates community preferences to reduce bias while improving minority-group model performance [1]. The paper, submitted on June 17, 2026, addresses a gap in current alignment research, which the authors say has “predominantly focused on predicting a unified response preference of annotators from certain regions” [1]. SCPO instead trains reward models to incorporate varied cultural preferences in a balanced manner [1]. The algorithm was tested on two datasets, PRISM and GlobalOpinionQA, across 7 countries [1]. Results show performance increases of the minority reward model of up to 7 points over the baseline model [1]. The researchers also report that SCPO is up to 280% more training data-efficient than full-data finetuning of reward models [1]. Bias mitigation is a central feature of the approach. The team performed analysis by separately evaluating on the preference of subcommunities and found that excessive bias was reduced via their weighting method [1]. This focus on equitable representation aligns with broader international frameworks. The United Nations’ 17 Sustainable Development Goals, adopted in 2015, include targets for reduced inequalities and peace, justice, and strong institutions [9]. A 2025 UN report noted that only 35% of SDG targets were on track or making moderate progress, with nearly half moving too slowly [9]. The development of SCPO also intersects with longstanding discussions in organizational ethics. Business ethics literature has long examined how norms and values guide conduct within institutions [4]. A 1990 article in the Journal of Business Ethics stated that “managing ethical behavior is one of the most pervasive and complex problems facing business organizations today” [4]. Applying such principles to algorithmic design, the SCPO method seeks to encode fairness directly into model training rather than treating it as an afterthought [1]. The code for SCPO has been made publicly available on GitHub, hosted under the repository minsik-ai/Steerable-Cultural-Preference [1]. The release allows other researchers to test and extend the method on additional datasets and cultural contexts.