GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

GroundEval is a new framework designed for evaluating AI agents based on deterministic tests of their evidence usage, addressing limitations in LLM-as-judge methods. It generates questions from a domain configuration, allowing agents to answer while scoring their final responses and the evidence retrieval process, targeting failures in checking evidence, reasoning from available information, and employing correct causal mechanisms. This framework provides structured diagnostics that reveal discrepancies between plausible answers and the actual evidence paths taken, which is crucial for practitioners to ensure the reliability and accountability of AI agents in real-world contexts.