Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

Researchers have proposed a multi-agent peer-review method that lets large language models evaluate each other’s reasoning to improve accuracy on medical exam questions, according to a preprint posted to arXiv on 13 June 2026 [1]. The approach assigns multiple LLM agents to independently generate chain-of-thought reasoning and candidate answers, then act as peer reviewers assessing each other’s outputs for factual correctness and logical soundness. The reasoning chain rated highest is selected to produce the final answer [1]. The study tested five state-of-the-art models — Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, and GPT-oss-20B — across three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA [1]. The peer-reviewed reasoning method consistently outperformed two baselines: single-model chain-of-thought reasoning and chain-of-thought-based majority voting. The best model combination reached an average accuracy of 0.820 across the datasets, compared with 0.777 for the strongest individual model and up to 0.789 for majority-voting ensembles [1]. The authors also reported that performance scaled with the number of participating models, and that peer assessments reliably separated high-quality reasoning chains from low-quality ones [1]. Medical question answering has drawn sustained interest as generative AI systems have advanced. Since the 2020s, generative AI — systems that create text, images, and other media from prompts — has become widely available, fueling an AI boom marked by rapid investment and public attention [4]. Chatbots such as ChatGPT, which reached 100 million monthly active users within two months of its November 2022 launch, have been praised for their potential to transform professional fields but also criticized for generating plausible-sounding yet incorrect answers, a phenomenon known as hallucination [3]. In biomedical domains, where factual errors carry direct risk, the new method’s emphasis on reasoning quality rather than answer agreement alone is designed to improve interpretability and robustness [1]. The preprint’s authors conclude that enabling LLMs to serve as both solvers and evaluators offers a direction for building more trustworthy biomedical AI systems [1].