Safety-Aware Evaluation of LLM-Generated Driver Intervention Messages through Multi-Task Risk Fusion

The paper introduces the Driver Safety-Aware Intervention Score (DSAIS), a novel metric designed to evaluate driver intervention messages generated by large language models (LLMs) based on five dimensions including risk-urgency alignment and driver acceptability. Utilizing a hybrid architecture that merges rule-based computation with LLM evaluations, the framework integrates multi-task recognition outputs, achieving high inter-rater reliability (ICC 0.798-0.840) and demonstrating a 9.1% improvement in contextual relevance over traditional rule-based systems. The findings highlight the effectiveness of compact local LLMs (7B-9B parameters) in enhancing intervention quality, offering insights for practitioners focused on developing adaptive in-vehicle AI systems.