Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers

The paper introduces an automated evaluation framework, adapted from the Honeyquest instrument, to assess the performance of 21 large language models (LLMs) as AI attackers against human deception. The models, ranging from 8 billion to over 1 trillion parameters, were evaluated on 174 reconnaissance queries, revealing that LLMs fell for deceptive traps at significantly higher rates than human attackers, lacked the defensive attention-diversion effect, and exhibited a critical recognition-action gap. These findings indicate that traditional human-centered deception strategies do not apply to AI attackers, emphasizing the necessity for developing AI-native active defense mechanisms.