An Empirical Study of Automating Agent Evaluation

The study presents EvalAgent, an AI assistant designed to automate the agent evaluation process, which traditionally requires extensive domain expertise. EvalAgent utilizes evaluation skills—comprising procedural instructions, reusable code, and dynamic API documentation—to create a trace-based pipeline that generates comprehensive evaluation artifacts. The introduction of the Eval@1 metric demonstrates significant improvements in execution success rates, increasing from 17.5% to 65%, and highlights the importance of domain-specific evaluation skills in enhancing the reliability of automated evaluations for AI practitioners.