Automated reproducibility assessments in the social and behavioral sciences using large language models

Large language models can automate reproducibility checks in the social and behavioral sciences, matching or exceeding human reanalysts on key metrics, according to a preprint posted to arXiv on June 11, 2026 [1]. The study, led by Stefan Feuerriegel, tested an LLM pipeline on 76 published studies with predefined claims [1]. For seven of those studies, the model could not produce a viable effect-size estimate [1]. Among the remaining cases, the LLM recovered the original effect sizes within a tolerance of ±0.05 in Cohen’s d for 41 percent of studies [1]. By comparison, human reanalysts recovered the original effect sizes in 34 percent of studies [1]. The LLM reached the same qualitative conclusion as the original paper in 96 percent of cases, while human reanalysts did so in 74 percent of cases [1]. The authors argue the results show LLMs can serve as a scalable tool for auditing empirical findings [1]. The work lands amid a broader push for open science, a movement to make research data, code, and publications transparent and accessible [6]. Traditional reproducibility checks rely on independent researchers reanalyzing original data, a process the paper describes as resource-intensive and difficult to scale [1]. Open-science advocates have long called for systematic auditing of published claims, but manual reanalysis has remained the default method [6]. The preprint appeared on arXiv, the open-access repository that hosts e-prints across physics, mathematics, computer science, and related fields [10]. arXiv submissions are moderated but not peer-reviewed, and the platform now receives roughly 24,000 articles per month [10]. The repository also supports community-built tools through its arXivLabs framework, which hosts experimental features such as citation explorers and recommender systems on article pages [9]. Those tools, including the CORE Recommender and Bibliographic Explorer, aim to help researchers discover relevant work and navigate citation networks [8][9]. The study’s findings add to a growing list of applications for artificial intelligence in scientific workflows. AI systems are already used for language translation, image recognition, and decision-making across industry and academia [5]. The new results suggest that LLMs could extend that footprint into the quality-control layer of published research, though the authors note that the pipeline failed to produce estimates for a small subset of studies [1].