GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science

A new evaluation environment called GRACE-DS aims to test large language model-driven AutoML agents under production-like conditions before deployment, according to a paper posted to the arXiv preprint server. [1] The framework, formally named the Guarded Reward-guided Agent Correction Environment in Data Science, subjects agents to a full machine-learning workflow inside an isolated setting. [1] The stages include planning, data inspection, feature engineering, model development, validation, code repair, and final submission. [1] Hidden executable validators then measure not only final predictive performance but also leakage avoidance, reproducibility, protocol validity, correction behavior, and reward alignment. [1] The paper, submitted on 14 June 2026 by Aleksandr Tsymbalov, reports that a structured regime called flexible iterative interaction outperformed single-shot generation, unstructured interaction, and restart-based baselines on end-to-end normalized hidden-test quality. [1] The same regime also improved protocol-valid completion rates. [1] The authors validated the approach across more than 7,000 episodes. [1] GRACE-DS is designed for tabular machine-learning tasks that are specific to a single organization, allowing firms to evaluate agents against their own data and compliance requirements before putting them into production. [1] The work appears on arXiv, an open-access repository that hosts electronic preprints across physics, mathematics, computer science, and related fields. [6] As of November 2024, the repository was receiving roughly 24,000 submissions per month. [6] Papers on arXiv are moderated but not peer-reviewed. [6] The submission, sized at 253 KB, is accompanied by experimental tools available through arXivLabs, a framework that lets community collaborators build features directly on the abstract page. [1][4] arXivLabs projects have included citation explorers, code-and-data linkers, and recommender systems, all governed by guidelines on openness and user-data privacy. [4][5] The arXiv team paused new Labs proposals while it modernizes infrastructure, though existing projects continue to operate. [3] Large language models, the type of system GRACE-DS evaluates, are machine-learning models with many parameters trained on vast text corpora for tasks such as language generation. [8] The new environment adds a layer of pre-deployment scrutiny that the authors argue is missing from current AutoML benchmarks. [1]