BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems

BELLS-O, the first independent operational benchmark for LLM supervision systems, evaluates 28 systems from 17 providers on metrics such as detection rate, false-positive rate, latency, and cost. It includes specialized guardrails like LlamaGuard-4 and generalist LLMs like GPT-5.4, assessing input/output moderation across 11 harm categories and jailbreak detection across 13 attack techniques. The findings indicate that specialized supervisors outperform generalist LLMs in content moderation, achieving similar detection rates at significantly lower costs and latencies, thus providing practitioners with a vendor-neutral framework for selecting effective safeguards in real-world AI deployments.