Counsel: A Meta-Evaluation Dataset for Agentic Tasks

The article introduces Counsel, a public meta-evaluation dataset designed for assessing agentic tasks, which includes critiques from open-weight LLM-based judges (LLMJs) on benchmarks such as tau-bench and DA-Code, alongside human meta-evaluations of these critiques. The dataset enables the calibration and enhancement of LLMJs by stratifying critiques based on human alignment and reasoning quality, achieving a Krippendorff's alpha of 0.78 for inter-annotator agreement. This resource is significant for practitioners as it provides a foundation for improving the reliability of automated evaluations in complex multi-step tasks, thereby facilitating better training and performance measurement of agentic systems.