CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

The paper introduces CORE-Bench, a benchmark designed to assess the computational reproducibility of AI agents in scientific research, comprising 270 tasks derived from 90 studies across multiple disciplines. It evaluates agents like AutoGPT and CORE-Agent using GPT-4o and GPT-4o-mini, with the top agent achieving only 21% accuracy on the most challenging tasks, highlighting significant room for advancement in automating scientific processes. This benchmark is crucial for enhancing reproducibility in research, enabling the development of more capable AI agents that can not only replicate but also innovate in scientific inquiry.