Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs

A new framework aims to correct a persistent flaw in how large language models are ranked against one another, proposing to filter out contradictory comparison results before a final leaderboard is assembled [1][2]. Pairwise evaluation, in which two responses to the same prompt are compared, has become a standard method for assessing large language models (LLMs) on open-ended tasks [1][2]. The resulting judgments are then aggregated into an overall ranking. However, this paradigm is vulnerable to intransitivity, where the comparisons produce cycles such as A preferred over B, B over C, and C over A, or inconsistencies involving ties [1][2]. These contradictions can make the resulting leaderboard unstable and difficult to interpret [1][2]. A paper submitted to arXiv on 16 June 2026 introduces a prompt perturbation framework designed to improve the consistency of these evaluations [1][2]. The approach generates perturbed variants of each prompt and uses the resulting comparison graphs to identify and filter out structurally inconsistent comparison patterns [1][2]. Standard ranking methods are then applied to the filtered comparisons [1][2]. The authors state that incorporating graph-level structural consistency explicitly into the evaluation pipeline before ranking aggregation provides a principled way to reduce cyclic inconsistencies [1][2]. The paper appears on arXiv, an open-access repository for electronic preprints that, as of November 2024, receives about 24,000 submissions per month [7]. The work is listed under the Computation and Language category [1]. The repository itself is not peer-reviewed, but it serves as a primary distribution channel in fields such as computer science and physics [7]. The research is surfaced through arXivLabs, a framework that allows community collaborators to develop and share experimental tools directly on the arXiv website [5][6]. arXivLabs projects, which include bibliographic explorers and recommender systems, operate under guidelines that require partners to share arXiv’s values of openness, community, excellence, and user data privacy [5]. The arXiv team has stated that third-party collaborators receive only minimal and anonymized data about users, strictly for ensuring the correct functioning of the Labs features [5]. LLMs are machine learning models with many parameters, trained with self-supervised learning on vast amounts of text for tasks such as language generation [9]. The proposed framework does not alter the underlying models but intervenes in the evaluation process itself, aiming to produce rankings that more reliably reflect model capabilities [1][2].