Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining

A new analysis of Classifier-based Quality Filtering (CQF), a widely used technique for cleaning large-scale pretraining datasets, finds that the method improves downstream task performance but does not necessarily enhance language modeling on the targeted high-quality data, according to a paper posted on arXiv [1]. The study, led by Louis Béthune and submitted on 1 Oct 2025, provides an in-depth examination of CQF, which trains a binary classifier to distinguish between general pretraining data and a smaller, curated high-quality set [1]. The classifier assigns a quality score to each document, and only the top-scoring documents are retained for training [1]. The authors report that while this process boosts performance on downstream tasks, it does not consistently improve language modeling on the high-quality dataset itself [1]. The paper explains this paradox by demonstrating that CQF implicitly filters the high-quality dataset as well as the low-quality one [1]. A particularly striking finding is that training on CQF-selected data can outperform training directly on the high-quality set, even when that set is sufficiently large [2]. This result is notable given the substantial effort and cost recently devoted to augmenting high-quality data [2]. The researchers further compared models trained with CQF to those trained on synthetic data of increasing quality, obtained via random token permutations, and found starkly different trends [1]. The results challenge the view that CQF captures a meaningful notion of data quality, and the authors introduce an optimization-driven definition of data quality that can be reliably estimated using small-scale proxy experiments [2]. The work appears on arXiv, an open-access repository of electronic preprints that, as of November 2024, receives about 24,000 submissions per month and hosts over two million articles [7]. The repository is a primary distribution channel in fields such as computer science and machine learning, where large language models are trained on vast amounts of text using self-supervised learning [9]. The challenge of curating high-quality training data is well-documented; high-quality labeled datasets are usually difficult and expensive to produce because of the large amount of time needed to label the data [3]. The paper’s findings add to the ongoing discussion about data selection methods, which are essential given that large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality [2].