How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks

This paper presents a paired noise-floor protocol for evaluating multi-agent LLM coordination by examining the performance of Claude Haiku 4.5 against tau^2-bench retail. The study reveals that significant benchmark differences reported in prior works may not hold under controlled conditions, as no significant trial-0 contrasts were found across multiple seeds, with observed gaps ranging from -3 to +18 percentage points. This highlights the need for rigorous replication and validation of coordination effects in LLM architectures, as many reported improvements fall below the observed noise floor, questioning their robustness in practical applications.