CareTransition-Audit: A Benchmark to Audit Discharge Summaries for Efficient Care Transitions

The article presents CareTransition-Audit, an automated framework for auditing discharge summaries using large language models (LLMs), operationalizing the DISCHARGED framework into a checklist of 46 questions. Benchmarking 11 LLMs on 50 clinician-annotated summaries from the MIMIC-IV database, the mean documentation completeness ranged from 54.9% to 74.2%, with the best models achieving a Cohen's kappa of approximately 0.5, indicating moderate agreement with clinician assessments. This benchmark highlights the limitations of current models in identifying ambiguous documentation, providing a foundation for quality improvement in clinical documentation practices.