Towards Engineering Scaling Laws with Pretraining Data Composition

Researchers have made advancements in understanding neural scaling laws and compressing Large Language Models (LLMs), according to two recent papers submitted to arXiv on June 18 and June 23, 2026^[1][2].

The first paper, submitted on June 18, explores how neural scaling laws apply to models in particle physics, showing that performance scales as a power law in compute, model size, and dataset size, similar to language models^[1]. High-fidelity simulators in particle physics can produce synthetic data cheaply, allowing for the pretraining dataset to be engineered to influence scaling behavior. The study demonstrates that by including more diverse and task-aligned pretraining data, the scaling behavior can be engineered to require more data rather than larger models. Meanwhile, the second paper, submitted on June 23, focuses on deriving empirical scaling laws for domain-specific LLM compression. It finds that in-domain task quality degrades predictably under compression, while general-knowledge benchmarks collapse earlier^[2]. LLMs achieve strong performance across various domains but their scale poses deployment challenges in applications with latency and cost constraints.