Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

The paper presents an evaluation of LLM-based dialogue systems, specifically GitHub Copilot, in the context of assessing Non-Functional Requirements (NFRs) related to HIPAA compliance. It identifies limitations in current benchmarks that focus on functional correctness, proposing new methods to evaluate multi-turn interactions based on requirement satisfaction, reasoning, and code localization. The study reveals a discrepancy between developer agreement with LLM outputs and low accuracy against expert assessments, highlighting the need for improved designs in LLM dialogue systems to enhance satisfaction and effectiveness in collaborative reasoning.