Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison

Headache specialists preferred summaries written by their peers over those generated by three advanced large language models, though they often struggled to tell the two apart, according to a new study that directly compared human and AI-produced medical literature syntheses [1]. The study, posted on arXiv, tasked ten headache specialists from the United States and Canada with evaluating summaries produced by a retrieval-augmented generation framework using Sonnet, GPT-4o, and Llama 3.1 [1][2]. A single headache specialist crafted 13 clinical questions, three of which were used to optimize the AI prompts and ten for the formal evaluation [1][2]. Each of the ten evaluating physicians wrote one expert summary, resulting in four summaries per question: one human-written and three AI-generated [1][2]. The specialists, blinded to authorship and barred from reviewing their own topic, scored each summary on a 1-to-10 scale across four dimensions: correctness, completeness, conciseness, and clinical utility [1][2]. They also ranked the summaries by overall preference and guessed whether each was written by a human or a machine [1][2]. Expert-written summaries received higher preference rankings, yet the evaluators found it challenging to reliably distinguish human from AI output [1][2]. The research team identified specific features valued by clinicians that fall outside standard evaluation metrics, which they say can inform the next generation of both human and automated literature summarization pipelines [1][2]. The findings arrive as clinicians face mounting pressure to stay current with a rapidly expanding body of published research while managing limited time with patients [2]. The study’s design reflects a broader push in medical informatics to subject large language models to rigorous, blinded human evaluation rather than relying solely on automated benchmarks [2].