MiniPIC: Flexible Position-Independent Caching in <100LOC

The article introduces Minimalistic Position-Independent Caching (MiniPIC), a lightweight design for vLLM that enhances retrieval-augmented workloads by utilizing a positional-encoding-free KV cache and user-controlled cache-reuse primitives. MiniPIC enables efficient caching with fewer than 100 lines of code changes, supports multiple caching methods like Block-Attention and Prompt Cache, and achieves a 49% improvement in prefill throughput on the 2WikiMultihopQA benchmark, while significantly reducing time-to-first-token for cached spans. This development is crucial for practitioners as it allows for flexible caching strategies without extensive server modifications, optimizing performance in AI inference tasks.