Skill Coverage: A Test Adequacy Metric for Agent Skills

The article introduces "skill coverage," a metric designed to evaluate the adequacy of testing agent skills in large language model agents by assessing whether observable behavior constraints from skill documents are exercised during agent trajectories. The metric reveals that existing benchmarks, when applied to SkillsBench, cover only 39.90% to 43.98% of documented skill behaviors, indicating significant gaps in testing despite task success. This metric is crucial for practitioners as it emphasizes the need for comprehensive evaluation of skill artifacts, beyond mere task completion, to ensure robust agent performance across diverse contexts.