CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

A new benchmark called CORE-Bench aims to measure how accurately AI agents can reproduce the results of published scientific studies, a task its creators describe as fundamental to the research process [1]. The benchmark, introduced in a paper posted to the arXiv preprint repository, consists of 270 tasks drawn from 90 scientific papers spanning computer science, social science, and medicine [1][2]. Each task falls into one of three difficulty levels and includes both language-only and vision-language challenges [1][2]. The arXiv repository, which hosts the paper, was founded in 1991 and now receives roughly 24,000 submissions per month, according to its Wikipedia entry [6]. Computational reproducibility — the act of recreating a study’s findings using its original code and data — remains a persistent hurdle across disciplines [1][2]. The CORE-Bench evaluation system is designed to measure agent accuracy in a fast, parallelizable manner, saving days of evaluation time per run compared to a sequential approach [1][2]. The researchers tested two baseline agents: the general-purpose AutoGPT and a task-specific system called CORE-Agent [1][2]. Both were evaluated using two underlying language models, GPT-4o and GPT-4o-mini [1][2]. On the hardest difficulty tier, the best-performing agent reached an accuracy of just 21%, a figure the authors say highlights substantial room for improvement in automating routine scientific work [1][2]. The paper’s corresponding author is listed as Zachary Siegel [1]. The work was submitted to arXiv on September 17, 2024, and revised on June 22, 2026 [1]. The authors argue that agents capable of reproducing existing research are a prerequisite for building systems that can conduct novel studies or verify the output of other research agents [1][2]. CORE-Bench arrives as the broader AI community continues to debate how to construct benchmarks that correspond to real-world tasks rather than artificial metrics [2]. The arXiv platform itself has long supported community-built tools through its arXivLabs framework, which allows third-party developers to create features such as citation explorers and code-finding utilities that sit alongside paper abstracts [4][5]. Those integrations operate under guidelines that require partners to uphold values of openness, community, excellence, and user data privacy [4].