Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning

A new auditing framework proposes to correct systematic biases in Large Language Models used as evaluators by treating the problem as positive-unlabeled learning, according to research posted on arXiv this month [1]. Large Language Models are increasingly deployed as automated judges for scalable evaluation, but these systems carry biases decoupled from semantic quality — verbosity bias being the most prominent [1]. Human supervision, while reliable for positive judgments, remains costly and selective, leaving the majority of outputs unlabelled and of uncertain quality [1]. The paper, titled "Quantifying and Auditing LLM Evaluation via Positive–Unlabeled Learning," reframes this challenge as a positive-unlabeled learning problem [1]. The authors propose a geometric auditing method called PUAudit, which operates in a fixed embedding space without retraining the judge or the underlying models [3]. The framework uses Partial Optimal Transport to align a small set of human-verified positives with a reliable subset of the unlabelled pool [2]. Unlike standard optimal transport, which forces all probability mass to be matched, Partial Optimal Transport explicitly allows only a fraction of mass to be transported [3]. This property accommodates contamination in the unlabelled set, preventing low-quality responses from being spuriously aligned with verified positives [3]. As a result, the method identifies plausible latent positives while remaining robust to noise and mismatch in the unlabelled distribution [4]. The approach shifts scalable LLM evaluation from a fully supervised labeling problem to a lightweight, representation-fixed auditing problem [5]. It yields a training-free procedure for recovering responses likely to be human-preferred from an unlabelled pool contaminated by low-quality generations [5]. Experiments reported in the paper demonstrate improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates [2]. The authors describe the work as a scalable and statistically grounded alternative to existing LLM-as-a-judge pipelines [1]. arXiv, where the paper was submitted on 17 June 2026, is an open-access repository of electronic preprints that are moderated but not peer-reviewed [9]. The repository hosts scientific papers across mathematics, physics, computer science, and related fields, with a submission rate of approximately 24,000 articles per month as of November 2024 [9].