Recency/Frequency Adaptive KV Caching for Large Language Model Serving

The article presents a novel adaptive key-value (KV) caching strategy for large language model inference, which improves cache management by dynamically allocating space between recently and frequently accessed KV blocks. This approach enhances the KV cache hit rate by up to 10.8% and reduces time to first token by up to 12.6% on synthetic workloads, and by 2.1% and 2.0% on real-world conversation tasks compared to naive vLLM. This advancement is significant for practitioners as it optimizes inference efficiency and accommodates diverse workloads, addressing limitations of traditional caching methods like least-recently-used (LRU).