The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge

The paper presents a framework for evaluating multi-agent debate systems by analyzing token-level log-probabilities, LLM-as-judge rubric scores, and final task accuracy. It employs a two-agent architecture consisting of a Constructor and an Auditor, revealing that confidence signals correlate more strongly with reasoning quality in the Constructor, achieving an AUROC of 0.804 for critical reasoning failure detection. This research is significant for practitioners as it emphasizes the importance of intermediate reasoning quality in multi-agent systems, potentially guiding improvements in LLM training and evaluation methodologies.