Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

New research finds that large language models used to evaluate AI-generated radiology reports exhibit a widespread discrimination bias, effectively catching clinical errors but over-penalizing harmless rephrasings, according to a paper submitted to arXiv in 2026 [1][2]. The study, posted on June 17, 2026, argues that existing metrics for evaluating radiology reports are medically ungrounded, reducing report quality to a single scalar score [1][2]. While LLMs possess rich medical knowledge, they struggle to draw a reliable boundary between clinically significant errors and harmless variation [1][2]. The researchers used the ReEvalMed benchmark as a testbed to study this boundary, evaluating metric-level clinical significance through two lenses: detecting true clinical errors, termed "Discrimination," and tolerating insignificant variations, termed "Robustness" [1][2]. Across eight LLM evaluators tested in both one-pass and two-pass settings, the team identified a consistent discrimination bias [1][2]. The models effectively detected errors but also over-penalized harmless rephrasings [1][2]. To mitigate this, the researchers synthesized 4,000 report pairs and trained lightweight interpretable metrics on Qwen3-8B and MedGemma-4B [1][2]. The resulting trained metric sharpened the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models [1][2]. The study also found that the more computationally costly two-pass setting failed to consistently improve overall performance and mainly traded discrimination for robustness [1][2]. These findings suggest one-pass trained metrics as the practical choice for cost-sensitive deployment, with two-pass inference reserved for settings where the discrimination-robustness balance is critical [1][2]. The authors stated they will release the dataset and metric [1][2]. The work addresses a critical gap in medical AI, where omitted critical findings or mischaracterized radiographic observations can directly affect patient care [1][2]. The paper was submitted through arXivLabs, a framework allowing collaborators to develop and share new features on the arXiv platform [1].