6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models

A new benchmark called AdversarialAnatomyBench reveals that vision-language models (VLMs) achieve a mean accuracy of only 28% on rare anatomical variants, compared to 71% on typical anatomy, according to a paper posted on arXiv [1]. The study, authored by Leon Mayer and colleagues, tested 25 state-of-the-art VLMs on basic medical perception tasks using naturally occurring rare anatomical presentations across diverse imaging modalities [1]. The benchmark introduces the concept of "natural adversarial anatomy" — variants that violate learned priors about typical human anatomy [1]. Even the top-performing models, GPT-5, Gemini 2.5 Pro, and Llama 4 Maverick, experienced performance drops of 41-51% when confronted with atypical cases [1]. The paper further reports that model errors closely mirrored expected anatomical biases, and that neither scaling up models nor applying interventions such as bias-aware prompting and test-time reasoning resolved the generalization failures [1]. The findings underscore a critical limitation in current multimodal medical AI systems, which are increasingly integrated into clinical workflows [1]. The research was posted on arXiv, an open-access repository that hosts electronic preprints across fields including computer science and quantitative biology and has grown to a submission rate of about 24,000 articles per month as of November 2024 [6]. The work provides a foundation for systematically measuring and mitigating anatomical bias in medical AI [1].