EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

EHR-Complex is a newly introduced benchmark aimed at evaluating interactive clinical database reasoning, built on the MIMIC-IV dataset, which includes 365K patients and over 500 million records. It consists of approximately 52,000 tasks across six clinical intents, requiring agents to execute SQL queries or Python code in a sandboxed environment, reflecting real-world complexities in EHR analysis. The benchmark reveals significant challenges in SQL task execution, with top models achieving only 62.3% accuracy and exposing failure modes such as SQL logic errors and semantic misunderstandings, underscoring the need for improved reasoning capabilities in clinical AI applications.