Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

The paper introduces "Metric Match," a novel method for evaluating the reliability of LLM judges using a subset selection approach that minimizes the need for extensive human annotations. This method achieves a win-rate of 0.838 against random selection across various correlation metrics and datasets, reducing average estimation error by 18.7% and annotation requirements by 32.5%. Practitioners can leverage Metric Match to enhance the efficiency of LLM evaluations, particularly in resource-intensive fields like healthcare, where it can significantly lower costs associated with expert annotations.