Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift

The study evaluates the performance of three prompt-injection detectors—ProtectAI-v2 and two Prompt-Guard-2 checkpoints—under conditions where the attack distribution shifts from the training benchmark. It introduces a severity metric (S) that measures the confidence of the detectors in the missed attacks, revealing that they maintain a severity score between 0.99 and 1.00 while exhibiting a high false-negative rate (0.01 to 0.97). This highlights a critical vulnerability in current detectors, as they can miss significant injection attacks with high confidence, emphasizing the need for improved calibration techniques in the development of robust AI security systems.