Vision-language models for chest radiography do not always need the image

A new audit of medical vision-language models finds that several systems achieve high accuracy on chest radiograph benchmarks without actually using the image, raising questions about whether standard accuracy metrics are sufficient to certify models for clinical deployment [1]. The study, posted to arXiv by Soroosh Tayebi Arasteh and colleagues, introduces a causal audit that intervenes on the input image in three ways: occluding the clinically relevant region, occluding an irrelevant region, and swapping in another patient's scan that carries the same diagnostic label [1]. The audit combines three behavioral metrics to determine whether a correct answer genuinely depends on the image [2]. Across nine systems tested, a text-only model with no access to the image reached within 5.7 accuracy points of the best multimodal model, and a 119-billion-parameter multimodal model proved statistically indistinguishable from a 7-billion text-only baseline [1][2]. Chest radiography relies on X-ray imaging, a technique discovered in 1895 by Wilhelm Conrad Röntgen that uses high-energy electromagnetic radiation to penetrate living tissue [4]. Modern medical AI systems are increasingly evaluated on their ability to interpret such images, but the authors argue that benchmarks currently in use do not separate models that exploit finding-name priors from those that actually read the scan [1]. The audit sorted the nine models into three categories: three that ignore the image entirely, one that is unstable, and five that use the image selectively for a subset of findings. These categories held across a second dataset, a different resolution, and alternative prompt phrasing [2]. When compared against board-certified radiologists, a text-only model was statistically indistinguishable from a radiologist's accuracy while grounding at zero — meaning it could not point to the image region supporting its answer. In contrast, the image-using models grounded at radiologist-comparable rates [1][2]. The study also found that reported confidence flags identified ungrounded answers only when a model actually used the image [2]. The findings carry implications for regulatory oversight of medical AI. CT scanning, developed in the 1970s and recognized with the 1979 Nobel Prize in Physiology or Medicine, has become a cornerstone of diagnostic imaging alongside conventional X-ray [3]. As AI systems are proposed for integration into these workflows, the authors recommend that grounding audits — not accuracy scores alone — should gate clinical deployment [1][2].