Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

The study published in arXiv investigates the efficacy of LLM-as-judge in evaluating multi-turn conversational agents, specifically a food-and-beverage ordering system. It reveals that the LLM judge identified only 22% of human-confirmed systematic problems and failed to flag operational issues in 100 rounds, highlighting a structured blind-spot in its scoring rubric, which inadequately addresses critical behavioral dimensions like state-tracking. This underscores the need for enhanced evaluation mechanisms in production environments, as reliance on automated judging can lead to significant underreporting of defects, emphasizing the necessity of human oversight in quality assurance.