LMCache separates KV cache management from inference into a dedicated process, achieving 14x faster time-to-first-token on H200s running Qwen3-235B. The open-source solution plugs into vLLM, SGLang, and TensorRT-LLM, sharing GPU memory directly between processes.
Key facts
- 14x faster time-to-first-token on H200s with Qwen3-235B.
- 4x faster decoding at 50 concurrent users.
- Startup drops from 3+ minutes to ~30 seconds.
- TurboQuant shrinks KV cache to 3 bits per value.
- TurboQuant reduces inference throughput by 20%+.
Every inference engine—vLLM, SGLang, TensorRT-LLM—makes the same architectural mistake: it runs KV cache management and token generation in the same process. As context windows grow, the cache competes with attention for GPU compute and memory bandwidth, forcing the two to take turns. According to @akshay_pachaar, that coupling means every improvement to caching costs inference throughput.
Google's TurboQuant illustrates the constraint. It shrinks the KV cache to 3 bits per value with no accuracy loss, so cached blocks take less GPU memory and travel faster to CPU RAM or disk. But compression and decompression run on the same GPU generating tokens. Switch TurboQuant on and inference throughput drops 20%+ compared to leaving it off. The compression is not the problem—the shared process is.
How the Split Works
LMCache moves cache management into a separate process running alongside the engine. Three things follow from that split:
Minimal handoff. The engine sends only block IDs—a few identifiers rather than gigabytes of data. All actual movement of cached vectors happens on the other side while the engine keeps computing attention.
Shared GPU memory. Passing cached state between two GPUs normally costs several copies of the data. LMCache processes read and write the same region, so a cached document stops being duplicated per worker.
Parallel lookups. Cached blocks live in four places: GPU memory, CPU RAM, local SSD, and cloud storage. Checking them one at a time means the slowest one sets your floor. LMCache queries all four at once and streams from whichever answers first.
That's the real shift. Cache work is I/O-heavy; inference is compute-heavy. Keeping them in one process means every improvement to caching costs you inference time. On H200s with Qwen3-235B and 50 concurrent users, separating them gives 14x faster time-to-first-token (TTFT) and 4x faster decoding. Startup drops from over three minutes to about 30 seconds.
LMCache plugs directly into vLLM, SGLang, and TensorRT-LLM, runs on NVIDIA and AMD, and survives a crash on either side. The engine falls back to uncached inference until the cache process reconnects. The project is open-source on GitHub.
What to watch
Watch for LMCache adoption in production vLLM deployments at scale, especially on AMD MI300X or NVIDIA H100/B200 clusters. If the architecture proves stable under high concurrency (500+ users), expect inference engine vendors to copy the split-process pattern by Q3 2026.
