MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

MMed-Bench-IR is a newly introduced benchmark for multilingual medical information retrieval, addressing the need for cross-lingual alignment, concept discrimination, and evidence retrieval across six languages. It comprises three tasks: cross-lingual medical QA retrieval with 6,127 queries based on the Unified Medical Language System, concept discrimination using 4,975 confusion sets, and multilingual evidence retrieval with 2,040 queries, all designed without overlap to accurately assess capabilities. The benchmark highlights significant performance gaps in biomedical encoders, with nDCG@10 scores dropping from 0.818 in English to 0.056 in Japanese, underscoring the limitations of existing English-only benchmarks for evaluating multilingual systems in clinical contexts.