From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents

The paper introduces KAPRO, a new benchmarking framework designed to evaluate the self-awareness capability of LLM agents by distinguishing between their metacognitive judgment and execution actions. It also presents the KAware dataset, which categorizes tasks into external, internal, and hybrid types to assess agents' cognitive alignment. The findings indicate that self-awareness is crucial for task success, with open-source models showing tendencies for tool overuse, which highlights the need for improved cognitive gating in LLM architectures.