Can LLMs Reliably Self-Report Adversarial Prefills, and How?

This study investigates the ability of large language models (LLMs) to recognize adversarial prefill attacks across ten instruction-tuned models ranging from 3B to 70B parameters. Findings indicate that no model consistently identifies its own compromised outputs, with an average self-report rate of only 27.3%. Various finetuning methods (SFT, GRPO, DPO) were tested, which increased the intention-probe gap but also inadvertently raised the success rate of adversarial attacks, underscoring the complexities and risks associated with LLM introspection in safety-critical applications.