MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworks

A new benchmark called MedicalAgentsBench pits internalized reasoning models against externalized agent-based frameworks on 862 complex clinical questions, finding that the two approaches are complementary and that combining them yields the highest accuracy reported to date. The benchmark, introduced in a paper first submitted in March 2025 and revised in June 2026, draws its questions from eight medical datasets using difficulty-aware curation and contamination screening [1][2]. Researchers evaluated three internalized reasoning models — DeepSeek-R1, o1-mini, and o3-mini — alongside seven base models and nine externalized agent-based methods [1][2]. Internalized reasoning refers to models that perform multi-step inference within a single call, while externalized agent scaffolding decomposes problems collaboratively among multiple large language models [2]. The study found that each approach independently improves performance, and that their benefits compound when layered together [1][2]. The highest accuracy, 35.1 percent, was achieved by pairing the o3-mini internalized reasoning model with the MDAgents agent framework [1][2]. A Pareto analysis showed that this combination dominates the cost-performance frontier, meaning it offers the best trade-off between expense and accuracy among the configurations tested [2]. The authors also note that lightweight optimization on inexpensive models provides an entry point for resource-constrained settings, broadening the potential applicability of the findings [2]. The benchmark and associated code are publicly available on GitHub under the Gerstein Lab repository [1][2]. The paper was authored by Xiangru Tang and collaborators [1].