Beyond Accuracy: Measuring Logical Compliance of Predictive Models

The paper introduces the Rule Violation Score (RVS), a new evaluation metric designed to measure the logical compliance of predictive models, alongside traditional accuracy metrics. RVS distinguishes between hard and soft rules, is applicable to any dataset and predictive model, and can be computed via automatically generated SQL queries for Horn rules. This metric is crucial for high-stakes applications, as it highlights logical consistency, potentially revealing significant differences in model behavior that conventional metrics may overlook.