SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters

The article introduces SAGA, a distributed scheduler designed for AI agent inference on GPU clusters, which shifts focus from individual LLM calls to program-level scheduling of entire agent workflows. SAGA implements three mechanisms: Agent Execution Graphs for predicting KV cache reuse, session-affinity batching with work stealing for load balancing, and the Agent Fair Share metric for fairness in task completion. In benchmarks on a 64-GPU cluster, SAGA achieved a 1.64x reduction in task completion time and improved GPU memory utilization by 1.22x, highlighting the importance of workflow-aware scheduling for optimizing latency in compound AI applications, despite a tradeoff in peak throughput.