Efficient Safety Benchmarking via Item Response Theory

The paper presents a novel approach to safety benchmarking for language models using Item Response Theory (IRT), which enhances the efficiency of evaluations by addressing the limitations of static paradigms. Key contributions include adaptive item selection that reduces evaluation costs by at least 80% while maintaining high correlation with full-benchmark rankings, and a method for creating a reusable subset of items that can save up to 99.8% on AIR-Bench 2024. This work is significant for practitioners as it offers strategies to streamline safety evaluations, improving resource allocation and model assessment accuracy in a landscape of heterogeneous safety items.