Alibaba Cloud production data shows 10% of KV cache blocks serve 77% of all hits. CacheBlend, from the LMCache team and winner of the EuroSys 2025 Best Paper Award, solves this by reusing per-document caches across any permutation.
Key facts
- 10% of KV cache blocks serve 77% of hits (Alibaba Cloud).
- CacheBlend achieves 2-4x faster multi-document processing.
- EuroSys 2025 Best Paper Award winner.
- Prompt caching yields 60-85% hit rates on stable prefixes.
- Cached input tokens get up to 90% cheaper.
Prompt caching is the highest-leverage optimization for stable system prompts and tool definitions, delivering up to 90% cost reduction on cached input tokens with 60-85% hit rates According to @akshay_pachaar. But it demands byte-for-byte prefix exactness—change one character, get a full cache miss. This breaks in three common scenarios: RAG with multiple documents, where caching document A and B separately means a query needing both invalidates B; document order changes, where every permutation is a miss; and growing conversation history, where earlier cached states become useless as the prefix shifts.
Key Takeaways
- CacheBlend reuses per-document KV caches by recomputing only boundary tokens, achieving 2-4x speedups on multi-document queries.
- Alibaba data shows 10% of blocks serve 77% of hits.
The 10% Problem
Alibaba Cloud's production data reveals the scale of waste: 10% of KV cache blocks serve 77% of all hits. The rest sits in storage, never reused, because prefix matching prevents it.
CacheBlend's Solution
CacheBlend, a research paper from the LMCache team (EuroSys 2025 Best Paper Award), attacks this by recognizing that in modern transformers, tokens overwhelmingly attend to their own local context. Only a small fraction of tokens carry real connections across document boundaries. Instead of recomputing everything after the first cached document, CacheBlend reuses every document's cache as-is and selectively recomputes just those few boundary tokens. The result is 2 to 4x faster processing on multi-document queries with no quality loss. Document order no longer matters—shuffle documents however you like, and every permutation stays cached, where prefix caching recomputes all of them every time.
From Prefix Caching to Knowledge Caching
This shifts the paradigm from caching prefixes to caching knowledge. Every document in your knowledge base becomes a reusable cached asset, regardless of order or adjacency. CacheBlend ships inside LMCache, an open-source cache management layer that runs outside the inference engine and integrates with vLLM, SGLang, and TensorRT-LLM on both NVIDIA and AMD GPUs.
What to watch
Watch for production adoption metrics of LMCache in vLLM and SGLang deployments over the next six months, and whether inference engine vendors integrate selective recomputation natively rather than relying on an external cache layer.
