Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

Litmus is a new zero-label system designed for the specification of evaluation metrics in AI pipelines, leveraging source code and targeted interrogation to derive evaluation intent. It was tested on three real AI applications—financial account grouping, scientific QA, and inherent risk assessment—where it outperformed existing baselines like AutoMetrics and DynamicRubric in terms of concern coverage and validity, achieving a Spearman correlation of 0.72 on scientific QA. This approach emphasizes the importance of understanding what needs to be measured and why, potentially transforming the process of metric specification in AI system evaluation.