GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution

A new method called GRASP aims to improve how researchers identify the most valuable training data for large-scale machine learning models by accounting for interactions between data points, according to a paper posted to the arXiv preprint server [1]. The paper, submitted on 5 June 2026, introduces GRASP, which stands for Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution [1]. The authors argue that current scalable data attribution methods are limited because they assign isolated utility scores to individual training examples, an additive assumption that fails to capture critical subset dynamics such as data redundancy and complementary coverage [2]. GRASP reframes the problem as subset-level counterfactual utility prediction and explicitly models subset interactions through a quadratic geometric penalty [2]. To operate at pretraining scale without hidden oracle tuning, the method couples low-dimensional feature sketches with a strictly finite lower-confidence bound selection protocol [2]. In subset-retraining evaluations, GRASP more than doubles the task-level rank correlation for counterfactual subset fidelity compared to existing scalable baselines, while reducing upfront artifact construction costs by nearly an order of magnitude [2]. The paper further reports that the scoring mechanism transfers to language model curation and cross-domain vision selection [2]. Large language models, which are neural networks trained on vast amounts of text for tasks such as generation and translation, are known to be sensitive to the quality of their training data [8]. The work appears on arXiv, an open-access repository of electronic preprints that, as of late 2024, receives about 24,000 submissions per month and is not peer reviewed [6]. The paper is accompanied by experimental tools from arXivLabs, a framework launched in 2020 that allows community collaborators to develop and share features directly on the article record page, such as the Bibliographic Explorer and CORE Recommender [5].