Mitigating LLM-based p-Hacking by Preregistering for the Next LLM

A team of researchers has proposed a protocol designed to curb p-hacking in studies that use large language models, by requiring scientists to preregister their experiments and then run them on the first eligible model released after the commitment date, according to a paper posted to arXiv on June 26, 2026 [1]. The protocol targets a growing vulnerability in computational social science and other fields where LLMs are used to generate, classify, or annotate data for downstream hypothesis tests. Because a researcher can tune prompts, decoding parameters, or output formats until a statistically significant result appears, the risk of false positives is high [1]. The authors propose that researchers finalize their procedure on currently available models, preregister the analysis plan along with a list of eligible future models, and then execute the confirmatory analysis on the first eligible LLM released after the preregistration. Since that model does not yet exist at the time of commitment, it cannot be gamed [1]. The paper evaluates the protocol on two tasks with known true values. Across 20 models from four providers and 11 LLM-analysis configurations, the approach would have blocked the successful transfer of a p-hack in 73.9% of cases in the first task and 72.7% in the second [1]. The researchers also stress-tested the protocol and found that mitigation remained substantial under several alternative scenarios [1]. In a preregistered experiment conducted as part of the study, seven configurations that had successfully hacked a prior model were tested on the first eligible model released afterward. The hacking failed to carry over in six of the seven configurations [1]. The work arrives as the use of LLMs in research pipelines accelerates and as the scientific community grapples with reproducibility challenges. The preprint was posted on arXiv, an open-access repository that has hosted more than two million e-prints since its founding in 1991 and now receives roughly 24,000 submissions per month [6]. The paper appears within arXiv’s Computation and Language category and is accompanied by community tools such as the Bibliographic Explorer and CORE Recommender, which are part of the arXivLabs framework for third-party innovation [4][5]. arXivLabs, launched in 2020, allows collaborators to build experimental features on top of the repository while adhering to arXiv’s values of openness, community, excellence, and user data privacy [5]. The framework is currently on hiatus for new proposals while the development team focuses on migrating arXiv’s systems to the cloud [3].