AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability

AdversaBench is introduced as an automated red-teaming pipeline for large language models (LLMs), utilizing a combination of five structured mutation operators and a three-judge confirmation process to evaluate model failures. Experiments with 45 seed prompts across reasoning, instruction-following, and tool use categories revealed that the effectiveness of mutation operators varies significantly, with instruction-following prompts requiring more iterations to achieve failure. Notably, adversarial prompts generated for Llama 3.1 8B demonstrated zero-shot transferability to Llama 3.3 70B, indicating that the identified vulnerabilities may reflect general behavioral patterns rather than specific model weaknesses, which is critical for practitioners aiming to enhance LLM robustness.