CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

CrossPool is a new serving engine designed for cold mixture of experts (MoE) models, addressing GPU memory inefficiencies by separating feedforward network (FFN) weights and key-value (KV) caches into distinct memory pools. This architecture allows for dynamic KV-cache allocation based on active demand while consolidating FFN weights across multiple models, significantly improving GPU memory utilization and supporting long-context requests. CrossPool demonstrates a performance improvement over existing KV-cache-based multi-LLM serving systems, achieving up to a 10.4x reduction in P99 tail latency, which is crucial for practitioners aiming to optimize resource allocation and response times in LLM deployments.