CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

Researchers have proposed a new benchmark, CareTransition-Audit, to test whether large language models can automate the auditing of hospital discharge summaries, a process that currently relies on manual review and does not scale [1]. The framework operationalizes the DISCHARGED clinical framework into a checklist of 46 questions and was tested on 50 summaries from the MIMIC-IV database with clinician ground-truth labels [1][2]. Eleven large language models were benchmarked [1]. Model-assessed mean documentation completeness ranged from 54.9% to 74.2%, and the best-performing models achieved Cohen’s kappa values around 0.5 against clinician labels, indicating only moderate agreement [1][2]. All models struggled to identify ambiguous documentation, a gap the authors describe as a key limitation in current automated auditing [1][2]. The work, led by Akshat Dasula, provides a clinician-validated benchmark and zero-shot baselines for systematic quality improvement in clinical documentation [1][2]. Incomplete or inconsistent discharge documentation is known to drive care fragmentation and avoidable readmissions, making scalable auditing tools a persistent need in patient safety [1][2]. Machine learning, a field concerned with developing statistical algorithms that learn from data and generalize to unseen tasks, underpins the large language model approaches used in the study [8]. The benchmark’s reliance on zero-shot evaluation means models were not fine-tuned on the specific checklist, a design choice that tests their out-of-the-box clinical reasoning [1][2]. The moderate kappa scores suggest that while automation shows promise, it is not yet reliable enough to replace clinician review, particularly when documentation is vague or incomplete [1][2].