ProbeLLM: Automating Principled Diagnosis of LLM Failures

ProbeLLM is a benchmark-agnostic automated probing framework designed to diagnose large language model (LLM) failures by utilizing a hierarchical Monte Carlo Tree Search approach. It enhances failure discovery by balancing global exploration of new failure regions with local refinement of recurring errors, while employing tool-augmented generation for verifiable test cases. This method provides a more comprehensive and interpretable view of model weaknesses, facilitating a shift from isolated failure analysis to structured failure mode identification, which is crucial for practitioners aiming to improve LLM robustness.