AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

AGENTSERVESIM is introduced as a hardware-aware simulator specifically designed for multi-turn LLM agent serving, addressing the complexities of stateful program execution that traditional simulators overlook. It features components such as a Program Orchestrator, Tool Simulator, Session-Aware Router, and KV Residency Model, allowing for comprehensive evaluation of serving policies with a reported accuracy of within 6% of real-system behavior on commodity CPUs. This tool is significant for practitioners as it facilitates controlled exploration of agent-serving strategies, optimizing performance without the need for extensive and costly hardware deployments.