Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations

The paper presents a framework for auditing public AI evaluation archives using Bayesian inference, highlighting the discrepancies in reported performance metrics across different benchmarks, such as LiveBench and Open LLM Leaderboard v2. It demonstrates that a terminal leaderboard can misrepresent system performance due to selective reporting and missing data, with empirical results indicating significant variances in time to reach performance ceilings (23.03 vs. 75.13). This work is crucial for practitioners as it emphasizes the importance of transparent evaluation protocols and the need for robust methods to validate AI performance claims, thereby improving the reliability of benchmarking practices in AI development.