Learning to Route Languages for Multilingual Policy Optimization

A new training framework called language-routed policy optimization (LRPO) aims to improve the multilingual performance of large language models by adaptively selecting which languages to use during reinforcement learning, according to a paper published on arXiv [1]. Large language models (LLMs), which are machine learning models with many parameters trained on vast amounts of text via self-supervised learning, are built on heterogeneous multilingual corpora [1][3]. However, standard policy optimization methods often implicitly limit each training question to a single response language or depend on a fixed dominant language for supervision [1]. The researchers propose LRPO, an online policy optimization framework that treats language as a selectable variable [1]. The method elicits multilingual rollouts for each training question and integrates their relative quality into preference-based policy updates, which the authors state increases the diversity and informativeness of training signals under a fixed rollout budget [1]. To adaptively determine which languages to explore during reinforcement learning, the team introduced a trainable language router formulated as a multi-armed bandit [1]. This router balances exploration of underutilized languages with exploitation of more informative ones [1]. The paper reports that extensive experiments show LRPO consistently improves multilingual performance, demonstrating that adaptive language routing enables effective cross-lingual knowledge exploitation for training [1]. The work focuses on a specific class of AI systems. A reasoning model, also known as a reasoning language model, is a type of LLM specifically trained to solve complex tasks requiring multiple steps of logical reasoning, such as logic, mathematics, and programming [4]. The researchers have released all resources associated with the LRPO framework on GitHub [1].