Efficient and Sound Probabilistic Verification for AI Agents

This article presents a framework for probabilistic verification of AI agents, addressing the limitations of existing deterministic policy enforcement methods. It introduces a sound approach based on distributionally robust optimization that computes upper bounds on policy violation probabilities, accommodating ambiguous state transitions. The framework outperforms previous methods on standard benchmarks for terminal and tool calling agents, enhancing the security-utility trade-off, which is crucial for practitioners developing secure AI systems in uncertain environments.