Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

A new method can distinguish whether a decline in a language model's performance comes from the model itself or from a silent change in the automated judge evaluating it, researchers report. The technique uses a fixed set of human-labeled examples as an anchor to resolve this ambiguity [1]. Continuous evaluation of large language model (LLM) products typically relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is alerted when the score drifts downward [1]. The problem is that the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores, making every drift alarm ambiguous between a worse product and a changed judge [1]. The proposed method resolves this by maintaining a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, combined with a second betting e-process on the judge-versus-human gap and a guard-window rule that returns a verdict of none, system, or judge [1]. The researchers prove the approach is anytime-valid and one-way identifiable, meaning only the judge can move the anchors [1]. A key design constraint is that the anchor process must out-run the main process it guards, ensuring process orthogonality [1]. In tests involving two real judge changes, a silent version bump was detected as judge drift in 60 out of 60 runs with zero judge-to-system misattribution [1]. A contaminating strict-prompt change was correctly attributed on 110 of 120 runs at a guard width of 300 [1]. By comparison, the industry-default rolling z-test produced false alarms on 75% of drift-free streams [1]. The experiments replicated on a second domain, TL;DR summarization, with no re-tuning; there the strict-prompt change shifted scores harder, causing the anchors to fire faster and attribution to reach 240 out of 240 [1]. The monitor operates at roughly 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime [1]. LLMs are language models with many parameters, trained with self-supervised learning on vast amounts of text [8]. The evaluation challenge addressed by this work is relevant across the industry, as companies such as DeepSeek and Alibaba Cloud continue to release open-weight models that require reliable benchmarking [7][9].