Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

Every Eval Ever introduces a unified schema and community repository for AI evaluation results, addressing inconsistencies in data formats and evaluation frameworks that hinder comparative analysis. The project features a standardized JSON document for representing evaluations, automatic converters for popular formats, and a crowdsourced database on Hugging Face encompassing 22,235 models, 2,273 benchmarks, and 31 evaluation formats. This initiative is significant for practitioners as it facilitates cross-community evaluation science, enhances reproducibility, and streamlines the process of comparing model performance across diverse benchmarks.