Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System

The article presents a deployment-centered evaluation framework for predicting query-level rejection risk in clinical large language models (LLMs) integrated into electronic health records. The authors developed a pre-response classifier that leverages deployment-specific context—such as provider type and department name—alongside query content, achieving an AUROC of 0.719 over 4.5 months of user feedback. This approach highlights the importance of contextual factors in enhancing user acceptance of LLM outputs, paving the way for improved guardrail mechanisms in clinical AI applications.