cache
30 articles about cache in AI news
KV Cache Quantization Silently Breaks Safety Alignment, Paper Shows
KV cache quantization silently breaks LLM safety alignment, with Mistral-7B losing 15.2% refusals at 1.03x perplexity. PCR diagnostic recovers up to 97% alignment in 35 GPU-minutes.
DeepSeek-V4 Hits 500K Context with 90% Less KV Cache via FlashMemory
DeepSeek-V4 achieves 500K context with 90% less KV cache via FlashMemory's lookahead sparse attention, keeping only 13.5% of cache in GPU memory without retraining.
How One Developer Achieved a 46:1 Context Cache Ratio to Manage 39 Projects
The key takeaway is that maximizing Claude Code's prompt cache through long, context-dense sessions is the most effective way to scale individual productivity across multiple projects.
MLX-VLM Adds Continuous Batching, OpenAI API, and Vision Cache for Apple Silicon
The next release of MLX-VLM will introduce continuous batching, an OpenAI-compatible API, and vision feature caching for multimodal models running locally on Apple Silicon. These optimizations promise up to 228x speedups on cache hits for models like Gemma4.
How Telemetry Settings Are Silently Costing You Cache Tiers (And How To Fix It)
A confirmed bug links telemetry settings to cache TTL; disabling telemetry defaults you to 5-minute cache, increasing costs. Use environment variables and hooks to mitigate.
Anthropic's Silent Cache TTL Cut
Claude Code's default cache TTL was silently reduced to 5 minutes on April 2, drastically increasing token costs. Use hooks and settings to mitigate the impact.
Microsoft's 'Compress-Thought' Cuts KV Cache 2-3x, Boosts Throughput 2x
A new Microsoft paper shows language models can learn to compress their reasoning steps on-the-fly, slashing memory use 2-3x and doubling throughput. Crucially, 15 percentage points of accuracy come from 'leaked' information in KV cache after explicit reasoning is erased.
Google's TurboQuant Compresses LLM KV Cache 6x with Zero Accuracy Loss, Cutting GPU Memory by 80%
Google researchers introduced TurboQuant, a method that compresses LLM KV cache from 32-bit to 3-bit precision without accuracy degradation. This reduces GPU memory consumption by over 80% and speeds up inference 8x on H100 GPUs.
Google's TurboQuant Cuts LLM KV Cache Memory by 6x, Enables 3-Bit Storage Without Accuracy Loss
Google released TurboQuant, a novel two-stage quantization algorithm that compresses the KV cache in long-context LLMs. It reduces memory by 6x, achieves 3-bit storage with no accuracy drop, and speeds up attention scoring by up to 8x on H100 GPUs.
arXiv Survey Maps KV Cache Optimization Landscape: 5 Strategies for Million-Token LLM Inference
A comprehensive arXiv review categorizes five principal KV cache optimization techniques—eviction, compression, hybrid memory, novel attention, and combinations—to address the linear memory scaling bottleneck in long-context LLM inference. The analysis finds no single dominant solution, with optimal strategy depending on context length, hardware, and workload.
DualPath Architecture Shatters KV-Cache Bottleneck, Doubling LLM Throughput for AI Agents
Researchers have developed DualPath, a novel architecture that eliminates the KV-cache storage bottleneck in agentic LLM inference. By implementing dual-path loading with RDMA transfers, the system achieves nearly 2× throughput improvements for both offline and online scenarios.
Claude Code's New Read Cache Blocks 8% of Token Waste Automatically
The claude-context-optimizer plugin v3.1 now actively blocks Claude from re-reading unchanged files, saving an average 8% of tokens per session.
UV_EXCLUDE_NEWER: The Environment Variable That Cuts Your uvx CI Time by
Set UV_EXCLUDE_NEWER with a date in your GitHub Actions cache key to avoid redownloading uvx tools each run, saving 40+ seconds of CI time per workflow execution.
Never Let the LLM Write the Joins
This article details a two-phase text-to-SQL pipeline: Phase A deterministically plans (intent, entity resolution, joins, RBAC) and Phase B executes with bounded LLM calls. The subject graph caches entity mappings lazily, and security is enforced before the model sees any schema.
Prism v1.8 Adds CLI, MCP Server, and SDKs — Here's How to Use Them with
Prism v1.8's MCP server gives Claude Code direct control over caches, budgets, and routing. Install it in 2 minutes and ditch the dashboard for terminal-based AI infrastructure management.
DeepSeek v4 Pricing Cuts 75%: $0.43/M Tokens In
DeepSeek v4 API pricing permanently cut 75% to $0.43/M input, $0.87/M output, enabled by 27% compute and 10% cache vs v3.2.
Claude Code Digest — May 01–May 04
CCmeter's cache-busting insights can slash your Claude Code costs by up to 40% instantly.
Claude Code Digest — Apr 28–May 01
CCmeter's cache-busting insights can cut your Claude Code costs by up to 40% instantly.
CCmeter: The Open-Source Dashboard That Reveals Exactly Why Your Claude
CCmeter parses Claude Code's local session logs to surface cache-busting patterns, cost leaks, and model-swap simulations. Free, local-first, zero telemetry.
RedParrot: Semantic Caching Speeds Up NL-to-DSL for Business Analytics by
Xiaohongshu researchers propose RedParrot, a framework that caches normalized structural patterns of natural language queries to bypass expensive LLM pipelines, achieving 3.6x speedup and 8.26% accuracy improvement on enterprise datasets.
The 270-Second Rule: How to Cut Claude Code API Costs by 90% with Smart
Anthropic's prompt cache has a 5-minute TTL. Orchestrator loops running faster than 270 seconds pay ~10% of full input token costs.
Atomic Chat's TurboQuant Enables Gemma 4 Local Inference on 16GB MacBook Air
Atomic Chat's new TurboQuant algorithm aggressively compresses the KV cache, allowing models requiring 32GB+ RAM to run on 16GB MacBook Airs at 25 tokens/sec, advancing local AI deployment.
mlx-vlm v0.4.4 Launches with Falcon-Perception 300M, TurboQuant Metal Kernels & 1.9x Decode Speedup
The mlx-vlm library v0.4.4 adds support for TII's Falcon-Perception 300M vision model and introduces TurboQuant Metal kernels, achieving up to 1.9x faster decoding with 89% KV cache savings on Apple Silicon.
Helium: A New Framework for Efficient LLM Serving in Agentic Workflows
Researchers introduce Helium, a workflow-aware LLM serving framework that treats agentic workflows as query plans. It uses proactive caching and cache-aware scheduling to reduce redundancy, achieving up to 1.56x speedup over current systems.
NVIDIA's Memory Compression Breakthrough: How Forgetting Makes LLMs Smarter
NVIDIA researchers have developed Dynamic Memory Sparsification, a technique that compresses LLM working memory by 8× while improving reasoning capabilities. This counterintuitive approach addresses the critical KV cache bottleneck in long-context AI applications.
Claude Code Digest — Jul 13–Jul 16
Claude Code is no longer being treated like a chat assistant: the winning pattern this week is deterministic hooks, policy gates, and verification layers wrapped around an agent that can now hit 80.8% SWE-Bench.
ServiceNow's SynthDocBench Teases Apart VLM Long-Context Failure Modes
ServiceNow releases SynthDocBench, a controlled synthetic benchmark for long-context visual document understanding that varies length, layout, modality, and reasoning to diagnose VLM failures.
Build a Self-Sustaining Claude Code Environment: The Complete 14-Part System
Build a self-sustaining Claude Code environment with 14 components: memory, skills, autonomy, guardrails, and monitoring. Connect them into a feedback loop where measurements flow back into memory. Use CLAUDE.md and hooks.
Airbnb Cuts LLM Eval From Weeks to a Day With Deterministic Caching
Airbnb cut LLM eval from weeks to a day with deterministic caching and micro adapters. The approach trains bug-fix patches in under an hour per GPU.
How to Build and Publish an MCP Server
Build an MCP server in ~600 lines of TypeScript with @modelcontextprotocol/sdk, using the X syndication endpoint trick, and publish to npm, Anthropic Registry, and Glama for maximum AI tool discoverability.