Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes

This study introduces the concept of decision fidelity in evaluating LLM-based user simulators for conversational AI, highlighting a significant gap in existing frameworks that focus solely on communicative fidelity. The authors analyze 2,790 real customer interactions with LLM sales agents, revealing a "disengagement deficit" where simulators misrepresent non-buyers' behaviors, leading to inflated engagement metrics and misleading training outcomes. This finding is crucial for practitioners as it underscores the need for more accurate simulation models that reflect genuine user decision-making processes to avoid overestimating the effectiveness of AI-driven sales agents.