AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?

The article introduces AOR-Bench, the first benchmark designed to evaluate over-refusal in Large Audio Language Models (LALMs), addressing the challenge of these models incorrectly rejecting benign queries deemed harmful in isolation. The benchmark features 3,000 pseudo-harmful audio samples across six categories and assesses 12 LALMs from major model families, revealing a prevalent issue of over-refusal. Additionally, it explores lightweight strategies like Chain-of-Thought and activation steering to mitigate this problem, highlighting the importance of refining safety mechanisms in audio processing applications.