PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

PlanBench-XL is a newly introduced interactive benchmark designed to evaluate the long-horizon planning capabilities of LLM tool-use agents across 327 retail tasks utilizing 1,665 tools. It includes a mechanism to simulate real-world unpredictability through blocked or failing tool functions, revealing that even advanced models like GPT-5.4 struggle significantly under these conditions, achieving only 11.36% accuracy in severe blocking scenarios. This benchmark is crucial for practitioners as it highlights the challenges of adaptive planning in complex tool ecosystems, emphasizing the need for improved robustness in agent design.