CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

CompressKV is a newly proposed framework for compressing key-value (KV) caches in long-context large language models (LLMs), specifically targeting GQA-based architectures. It introduces the concept of Semantic Retrieval Heads (SRHs) to selectively retain critical tokens based on their semantic importance, significantly improving resource efficiency. In experiments, CompressKV maintained over 97% of full-cache performance using only 3% of the KV cache on LongBench and achieved 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack, highlighting its potential for optimizing memory usage in LLM inference.