Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study

A pilot study posted to the arXiv preprint server examines whether AI agents can distinguish visually confounded diseases without prior training, introducing a multi-agent framework that improved accuracy on one task but fell short of clinical reliability [1][2]. The paper, submitted by Zihao Zhao and revised on 18 Jun 2026, targets two imaging-only proxy tasks: melanoma versus atypical nevus, and pulmonary edema versus pneumonia [1][2]. In both pairings, visual features overlap substantially even though clinical management differs [2]. The authors benchmark representative agents in a zero-shot setting, meaning the models receive no task-specific fine-tuning before evaluation [2]. They then propose a contrastive-adjudication framework in which multiple agents deliberate to reach a diagnosis [2]. On dermoscopy data, the framework yielded an 11-percentage-point gain in accuracy and reduced unsupported claims on qualitative samples [2]. Despite the improvement, the authors state that “overall performance remains insufficient for clinical deployment” [2]. They also note that inherent uncertainty in human annotations and the absence of real-world clinical context further limit translation to practice [2]. The study appears on arXiv, an open-access repository that hosts preprints across physics, computer science, and related fields and that passed the two-million-article mark by the end of 2021 [6]. The platform’s arXivLabs program, which allows community collaborators to build experimental tools on article pages, operates under guidelines emphasizing openness and user-data privacy [4][5]. The paper’s first submission on 26 Feb 2026 was 564 KB; the revised version on 18 Jun 2026 grew to 768 KB [1]. The research contributes to a growing body of work on multimodal large language models, which are trained on vast text corpora and have recently been extended to agent-based systems [2][8].