One Probe Won't Catch Them All: Towards Targeted Deception Detection

The paper discusses the limitations of linear probes for deception detection in AI systems, highlighting that a single universal probe only achieves a modest improvement of +0.032 AUC. In contrast, when probes are tailored to specific types of deception, the potential performance can increase to +0.108 AUC, emphasizing the importance of matching probes to particular threat models. This research suggests that practitioners should focus on customizing their deception detection strategies rather than relying on a one-size-fits-all solution, as the choice of instruction pairs significantly influences probe efficacy.