AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows

A new evaluation framework called AIPatient Arena assesses large language models across eight dimensions of clinical competence using electronic health record data, finding persistent weaknesses in diagnostic accuracy and handling of ambiguous patient responses despite strong performance in interview skills and professional conduct, according to a preprint posted to arXiv on June 16, 2026 [1]. The framework integrates EHR data into patient-specific knowledge graphs to enable multi-turn physician-patient interactions, moving beyond static, single-turn evaluations that dominate current medical AI benchmarks [1]. Researchers applied AIPatient Arena to a primary cohort of 437 patients and two out-of-distribution validation cohorts of 119 and 67 patients [1]. LLMs scored well in medical interview questioning skills, with mean scores ranging from 4.43 to 4.99 out of 5, and in ethical and professional conduct, where scores ranged from 4.38 to 4.93 [1]. Clarity and transparency of clinical explanations also registered solid performance, with mean scores between 3.80 and 4.72 [1]. Performance dropped in information integration, where scores ranged from 3.19 to 4.21, and in medication safety and justification, which saw scores between 3.13 and 3.78 [1]. The weakest results emerged in three areas: handling of ambiguous patient responses scored 2.57 to 3.32, information coverage scored 2.08 to 3.02, and diagnostic accuracy and reasoning scored 2.63 to 3.55 [1]. Process-based evaluation revealed recurrent interaction failures, including repetitive questioning, omission of past medical history, and inadequate handling of uncertainty [1]. Richer conversational context improved diagnostic reasoning but yielded limited gains in treatment planning [1]. The findings indicate that final-answer accuracy alone is insufficient for evaluating clinical readiness and highlight the importance of assessing how models gather, interpret, and communicate information throughout a consultation [1]. The paper was submitted to arXiv, an open-access repository of electronic preprints that is not peer-reviewed and has hosted scientific papers since August 14, 1991 [6]. As of November 2024, the repository receives about 24,000 submissions per month [6]. The preprint appears under the Computation and Language category and includes integration with arXivLabs, a framework launched in 2020 that allows community collaborators to develop experimental tools on the platform [4]. arXivLabs projects operate under guidelines requiring partners to share arXiv's values of openness, community, excellence, and user data privacy [4].