RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories

A new method called RegMix-D improves how data mixtures are selected during large language model pretraining by using proxy training trajectories instead of only endpoint losses, according to a paper posted to arXiv on June 17, 2026 [1][2]. The paper, authored by researchers in computation and language, proposes RegMix-D as an extension of the existing RegMix method. While RegMix fits a regression model on small-scale proxy runs to select a single static data mixture, RegMix-D leverages the full loss trajectories produced during those proxy runs to predict optimal mixtures at multiple training stages [2]. The approach supports two deployment modes: an offline variant that generates a complete mixture schedule before target training begins, and an online variant that adapts the mixture during training using observed loss [2]. Experiments were conducted on 25 billion tokens from the Pile dataset using a 1-billion-parameter target model [2]. RegMix-D consistently outperformed both RegMix and DoReMi across 13 downstream tasks [2]. The method also demonstrated proxy efficiency: it surpassed RegMix even when using only 128 proxy models, which represents 25 percent of RegMix’s proxy compute budget [2]. The Pile is a publicly available dataset commonly used for language model research [2]. Large language models are machine learning systems with many parameters, trained with self-supervised learning on vast amounts of text for natural language processing tasks [8]. The paper appeared on arXiv, an open-access repository of electronic preprints that is moderated but not peer reviewed [6]. As of November 2024, arXiv received about 24,000 article submissions per month and had surpassed two million total articles by the end of 2021 [6]. The repository hosts papers across mathematics, physics, computer science, and related fields [6]. The RegMix-D paper’s abstract page includes links to community-developed tools through arXivLabs, a framework that allows collaborators to build experimental features on the site [4]. arXivLabs projects include bibliographic explorers, code finders, and recommender systems, all developed under guidelines that emphasize openness, community, and user data privacy [4][5].