Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System

The article presents an evaluation of LAMBDA, a multi-agent data-analysis system, utilizing a three-layer human-AI grading cascade on 153 numerical QRData tasks from DSGym. The grading system achieved 100% precision with the strict grader and a 97% recall for the lenient grader against human labels, demonstrating effective strategies for distinguishing genuine outputs from grading artifacts. This work is significant for practitioners as it highlights the importance of hybrid grading approaches and the impact of iterative nudging on grading success, which can enhance the reliability of automated assessments in complex data analysis tasks.