Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance

PhysAssistBench is a newly introduced benchmark designed for evaluating interactive assistance in clinical settings, integrating doctor-patient interactions with EHR systems. It utilizes real MIMIC-IV cases to create agentic patients, facilitating multi-turn clinical scenarios while maintaining factual accuracy. Experimental results indicate that current LLMs struggle with reliable assistance due to the need for coordinated capabilities across clinical knowledge, communication, and EHR system interactions, highlighting significant challenges for practitioners developing medical LLM applications.