SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models

A team of researchers has introduced SrDetection, a self-referential framework designed to identify data leakage in code large language models (LLMs) without requiring access to proprietary training data or manually tuned thresholds [1]. The framework, detailed in a paper submitted to arXiv on June 29, 2026, addresses a persistent challenge in the evaluation of code LLMs: benchmark scores artificially inflated by prior exposure to test data during pre-training [1][2]. Existing detection methods often rely on brittle heuristics like timestamp filtering or require external reference sets with non-generalizable thresholds [2]. SrDetection operates by generating semantically equivalent variants of a benchmark sample and then contrasting the model’s behavior on the original against its variants. It flags cases where the original is disproportionately easier for the model, signaling potential leakage [1][2]. The approach is designed to function in both gray-box settings, where model logits are accessible, and black-box settings, where only outputs are available [2]. In a controlled leakage detection testbed, SrDetection improved the average F1 score by 21.52 points in the gray-box setting and by 14.46 points in the black-box setting over strong baselines [1][2]. The researchers then applied the framework in a gray-box study of 15 widely used code LLMs across four popular benchmarks, uncovering benchmark-specific leakage patterns that go beyond what prior overlap-based analyses had revealed [1][2]. The paper’s abstract notes that the method demonstrates “robust, threshold-independent leakage detection” [2]. The research was posted on arXiv, an open-access repository for electronic preprints that, as of November 2024, receives about 24,000 submissions per month [6]. The platform, founded in 1991, allows researchers to disseminate findings rapidly before formal peer review [6]. The paper’s source code and data have been made publicly available through a linked repository [2].