RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments

RetailBench is a newly introduced benchmark designed to evaluate the long-horizon decision-making capabilities of large language model (LLM) agents in realistic retail environments, specifically focusing on single-store supermarket operations over a thousand-day simulation. The benchmark assesses LLMs on various operational tasks, including pricing and inventory management, over a 180-day evaluation period, revealing that only a few models can sustain coherent decision-making, with all LLMs lagging behind an oracle policy in terms of net worth and sales outcomes. This benchmark is significant for practitioners as it highlights the challenges LLMs face in maintaining strategic stability in dynamic environments and provides a framework for developing more reliable autonomous decision-making systems.