BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents

BioMedArena is an open-source toolkit designed to enhance the reproducibility of deep research agents in biomedical applications by standardizing evaluation processes across 166 benchmarks and 75 tools. It decouples six layers of evaluation, allowing for easy integration of new models with minimal engineering effort, and includes six agent harnesses and context-management strategies that improve performance across 12 backbones, achieving an average improvement of 15.01 percentage points over prior state-of-the-art results on eight benchmarks. This toolkit is significant for practitioners as it streamlines the evaluation and comparison of biomedical AI models, facilitating more consistent and reliable research outcomes.