Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

A new framework called the Holistic Data Scheduler (HDS) uses multi-objective reinforcement learning to optimize data mixing during large language model pre-training, according to a paper submitted on 23 Jun 2026 [1]. The approach formulates data scheduling as a continuous control problem and uses the Soft Actor-Critic algorithm [1]. The composition and mixing strategy of training data is a cornerstone of Large Language Model (LLM) pre-training [1]. Online Data Mixing (ODM), which adaptively adjusts data mixtures during training, has emerged as a method to improve efficiency, but existing approaches are limited by a single optimization perspective [1]. The HDS framework addresses this by integrating three reward functions: a data-driven reward for quality, a loss-driven reward capturing inter-domain influence, and a model-driven reward based on weight norms [1]. In systematic experiments on LLMs of various sizes, HDS reached the final validation perplexity of the next best method with 44% fewer training iterations on The Pile benchmark [1]. The framework also achieved a 7.2% improvement on the MMLU 0-shot task and consistent gains on other benchmarks [1]. The authors state the design enhances both training efficiency and final model capability [1]. The use of reinforcement learning for training data selection builds on broader efforts to make model development more resource-efficient. The computational cost of pre-training large models has drawn scrutiny, and techniques that reduce required iterations directly lower energy consumption and associated hardware demands. While the HDS paper does not explicitly frame its work in sustainability terms, the 44% reduction in training iterations represents a measurable efficiency gain that aligns with industry efforts to reduce the environmental footprint of AI development. Data mixing strategies have been explored in other scientific domains. For instance, researchers have investigated whether datasets are complementary for training machine learning models in catalysis informatics, using transfer learning to improve performance when consolidating data from multiple computational methods [5]. The Open Catalyst 2020 (OC20) dataset enabled the catalysis community to use transfer learning to improve model performance on smaller datasets, and subsequent work explored the extent to which OC20 could aid the OC22 dataset through transfer learning or joint training [5]. These cross-domain parallels illustrate the broader relevance of data composition strategies beyond language modeling.