The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

A new study finds that using large language models as automated judges—a practice now common for ranking AI outputs and training reward models—suffers from significant run-to-run inconsistency and position bias, raising questions about the reliability of single-trial evaluations [1][2]. The research, posted to arXiv, examined two OpenAI judge models, GPT-4o-mini and GPT-4.1-mini, across 29 tasks in 10 categories [1][2]. When presented with identical pairwise comparisons 50 times, the judges’ preferences flipped on average 13.6% of the time [1][2]. For 28% of the questions, the flip rate exceeded 20%, and one question reached a 56% flip rate [1][2]. GPT-4o-mini also showed a pronounced tendency to favor the first answer it saw, selecting option A 72% of the time, a bias the authors report as statistically significant with a p-value of 0.024 [1][2]. The study further identified a gap between pairwise and pointwise scoring. When judges assigned scalar scores on a 10-point scale, the mean gaps between competing answers were small, ranging from 0.19 to 0.36, and were not statistically significant in aggregate [1][2]. Yet the same judges frequently declared a winner in head-to-head matchups, suggesting they express confidence in a choice even when their own numerical ratings provide little evidence of a meaningful quality difference [1][2]. Beyond within-model instability, the two judges agreed with each other only 76% of the time, corresponding to a Cohen’s kappa of 0.51 [1][2]. Changing the wording of a prompt to a semantically equivalent template altered the majority outcome in 25% of tested cases [1][2]. Using deterministic decoding reduced but did not eliminate the inconsistency [1][2]. The authors constructed a reliability curve to estimate how many repeated trials are needed for a majority vote to match a 50-trial reference verdict with 95% probability. On average, 11 trials sufficed, but for high-variance questions the number rose to 15 [1][2]. The findings suggest that single-trial LLM judging is often too noisy for high-stakes evaluation and that multi-trial aggregation, position randomization, and explicit uncertainty reporting should become standard practice [1][2]. The work comes as the field of AI safety grapples with the challenge of ensuring systems behave as intended, a concern that has drawn increased attention since the rapid advances in generative AI that followed OpenAI’s release of ChatGPT in late 2022 [5][7]. OpenAI, founded as a nonprofit in 2015 and now structured around a for-profit public benefit corporation, has seen roughly half of its then-employed AI safety researchers depart by 2024, citing a deprioritization of safety goals [5]. The study’s authors note that because both judges came from a single provider, cross-provider replication remains an important next step [1][2].