LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation

A preprint submitted to arXiv on 14 Jun 2026 proposes a new protocol for evaluating large language models used as judges, finding that some models exhibit a “dark current” of spurious preferences even when given identical inputs [1]. The paper argues that LLM-as-a-judge systems, which are increasingly used to evaluate open-ended model outputs because human annotation is costly, slow, and difficult to reproduce, should be treated as measurement instruments rather than reported as simple accuracy or win-rate devices [1][2]. The authors introduce a Judge Datasheet protocol that quantifies several performance characteristics, including dark current under true-vacuum inputs, stable cross-sensitivity to surface variation, positional false preference, and target sensitivity on a controlled quality ladder [2]. In a case study of three open-weight models, the researchers found distinct behavioral profiles. Llama-3.1-8B showed high dark current and presentation-conflicted behavior, while Qwen2.5-14B was vacuum-clean and target-sensitive but mixed stable and positional over-discrimination [2]. Qwen2.5-32B was vacuum-clean with low stable cross-sensitivity and low positional false preference [2]. The protocol also examines how tie instructions affect judge behavior. A strict tie criterion eliminated Qwen2.5-32B’s false preference on equal-quality pairs but absorbed marginal target signals into ties while preserving sensitivity to larger quality differences [2]. The authors note that prompting moves the criterion, not the resolution, and they do not claim that the downstream mechanism hypothesis motivating the work is confirmed; the contribution is a metrological protocol for measuring the measuring device before downstream claims are made [2]. The work appears amid broader efforts to standardize evaluation in machine learning. Other recent preprints have explored transfer learning across computational datasets and the consolidation of anthological dataset collections for catalyst informatics, reflecting a wider push toward rigorous benchmarking practices [4].