AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming

AutoSpec is a framework designed to enhance safety rules for large language model (LLM) agents through counterexample-guided inductive synthesis (CEGIS) using inductive logic programming (ILP). It iteratively refines expert-designed safety rules based on user annotations, achieving an F1 score of 0.98 and 0.93 across two domains, with a 94% reduction in false positives while maintaining high recall. This approach is significant for practitioners as it produces interpretable, auditable rules that effectively balance safety and operational flexibility in LLM applications.