Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA

The article introduces SearchFireSafety, a new benchmark designed for evaluating structure-aware retrieval and safety in statute-centric legal question answering (QA). It focuses on fire-safety regulations and assesses models' abilities to retrieve hierarchically linked evidence while managing hallucination risks when context is incomplete. Experiments demonstrate that while graph-guided retrieval enhances performance, it also increases hallucination rates in domain-adapted models lacking essential statutory context, underscoring the importance of benchmarks that address both retrieval effectiveness and model safety in legal AI applications.