Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

A new study finds that large language models used as automatic judges flip their preferences 10.7 to 14.4 percent of the time when evaluation materials are presented in Chinese or a mix of Chinese and English instead of English alone, raising questions about the reliability of multilingual AI evaluation pipelines. The research, submitted in 2026, introduces Judge-LS, a lightweight meta-evaluation protocol that transforms response-pair items from the LLMBar benchmark into English, Chinese, and Chinese-English language-switched variants [1]. Four API-accessible judges were tested on the full 419-item LLMBar benchmark, yielding 13,408 successful pairwise judgments [2]. Across all models, Chinese and language-switched presentations induced 10.7–14.4% preference flips relative to English, and every judge achieved its highest accuracy in English [3]. Judge-LS operates on a simple principle: a reliable judge should preserve its preference under label-preserving language transformations and should not favor one language when two answers are translation-equivalent [5]. The protocol requires no model training, uses only API calls, and is feasible on modest local hardware [2]. The authors added confidence intervals, paired significance tests, and an automatic transformation audit with a sensitivity analysis that excludes mechanically flagged high-risk variants [3]. The findings align with broader concerns about language bias in LLM-as-a-judge systems. A separate study examining pairwise judging across language families found significant performance disparities, with European languages consistently outperforming African languages in same-language comparisons [4]. That work also observed that most models favor English answers in inter-language judging, and that answer language plays a more dominant role than question language in driving this bias [4]. Despite the preference flips, the Judge-LS study did not find evidence of a systematic English preference when judges evaluated translation-equivalent tie probes. Most probes were judged as ties, and non-tie decisions more often favored Chinese [5]. The authors suggest the central risk is not simply that judges prefer English; rather, judges can be sensitive to language presentation in ways that change correctness and interact with position bias [3]. Translation may alter perceived specificity, fluency, reasoning traces, or formatting rather than triggering a blind English reward [5]. Related research on morphologically rich languages reinforces the instability finding. A controlled cross-lingual evaluation using synthetic customer-support dialogues in Estonian, Finnish, and Hungarian found that pragmatic judgments such as coherence and instruction-following exhibited rank inversions and near-zero correlations across languages, even when generation conditions were held constant [7]. That study concluded that zero-shot judge transfer is unreliable for discourse-level assessment in such languages, motivating language-specific calibration against targeted human baselines [7]. The Judge-LS protocol is inexpensive, training-free, and easy to extend, making it a practical diagnostic for multilingual evaluation pipelines [3]. The experiment’s results remain visible after excluding mechanically flagged high-risk transformations, and paired tests show significant strict-correctness changes [5].