CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

CheXpercept has been introduced as a benchmark for evaluating expert-level lesion perception in chest X-rays, addressing limitations in existing vision-language models (VLMs) that focus primarily on disease-presence classification. The benchmark includes a dataset of 10,400 QA items from 2,100 CXRs, covering seven critical pulmonary and cardiac lesions, and evaluates 14 VLMs, revealing that performance significantly drops on fine-level and semantic tasks, with medical VLMs showing no substantial advantage over general models. This benchmark is crucial for practitioners as it emphasizes the need for improved model capabilities in clinical contexts and will be publicly available for further research.