caching
30 articles about caching in AI news
Airbnb Cuts LLM Eval From Weeks to a Day With Deterministic Caching
Airbnb cut LLM eval from weeks to a day with deterministic caching and micro adapters. The approach trains bug-fix patches in under an hour per GPU.
Continuous Semantic Caching
Researchers propose a theory-grounded semantic caching system that treats user queries as points in a continuous embedding space, using dynamic ε-net discretization and kernel ridge regression to cut inference costs and latency without switching overhead.
Google's Memory Caching Bridges RNN-Transformer Gap with O(NL) Complexity
Google's 'Memory Caching' method saves RNN memory states at segment boundaries, allowing tokens to reference past checkpoints. This O(NL) approach significantly improves RNN performance on recall tasks, narrowing the gap with Transformers.
Semantic Caching: The Key to Affordable, Real-Time AI for Luxury Clienteling
Semantic caching for LLMs reuses responses to similar customer queries, cutting API costs by 20-40% and slashing response times. This makes deploying AI-powered personal assistants and search at scale financially viable for luxury brands.
RedParrot: Semantic Caching Speeds Up NL-to-DSL for Business Analytics by
Xiaohongshu researchers propose RedParrot, a framework that caches normalized structural patterns of natural language queries to bypass expensive LLM pipelines, achieving 3.6x speedup and 8.26% accuracy improvement on enterprise datasets.
Unsloth × NVIDIA Cut LLM Fine-Tuning ~25% — Three Glue-Code Wins on Blackwell
Daniel & Michael Han at Unsloth, in collaboration with NVIDIA, published a joint guide quantifying three glue-code optimizations that combine for ~25% faster LLM training on B200 Blackwell hardware. The wins target overhead around the main kernels — caching packed-sequence metadata, double-buffered gradient checkpoint reloads, and a cheaper GPT-OSS MoE router using argsort + bincount. All three are merged via public PRs.
ML-Master 2.0 Hits 56.44% on MLE-Bench in 24-Hour Agentic Science Run
Researchers from Shanghai Jiao Tong University demonstrated ML-Master 2.0, an autonomous research agent that operated continuously for 24 hours on the MLE-Bench, achieving a 56.44% medal rate. The breakthrough centers on Hierarchical Cognitive Caching for state management, not reasoning, enabling long-horizon scientific workflows.
MLX-VLM Adds Continuous Batching, OpenAI API, and Vision Cache for Apple Silicon
The next release of MLX-VLM will introduce continuous batching, an OpenAI-compatible API, and vision feature caching for multimodal models running locally on Apple Silicon. These optimizations promise up to 228x speedups on cache hits for models like Gemma4.
Helium: A New Framework for Efficient LLM Serving in Agentic Workflows
Researchers introduce Helium, a workflow-aware LLM serving framework that treats agentic workflows as query plans. It uses proactive caching and cache-aware scheduling to reduce redundancy, achieving up to 1.56x speedup over current systems.
LMCache Splits KV Cache From Inference, 14x Faster TTFT on H200s
LMCache separates KV cache management from inference into a dedicated process, achieving 14x faster TTFT on H200s with Qwen3-235B at 50 concurrent users.
Anthropic Ships Claude Opus 5: Fable-Level Intelligence at Half the Price
Anthropic released Claude Opus 5 on July 24 with a 1M token context, 128k output, and Fable-5-approaching intelligence at half the price, unchanged from Opus 4.8.
CacheBlend: 2-4x Faster KV Cache for Multi-Doc Queries
CacheBlend reuses per-document KV caches by recomputing only boundary tokens, achieving 2-4x speedups on multi-document queries. Alibaba data shows 10% of blocks serve 77% of hits.
7 AI Agent Cost Optimization Strategies That Cut LLM Bills by Up to 90%
The source outlines seven cost optimization strategies for AI agents, including prompt compression and model routing, that can reduce LLM bills by up to 90%. This matters for retail and luxury brands deploying AI at scale where inference costs can become prohibitive.
Stop Prompting, Start System Building
Move from prompting to system-building with Claude Code. Use CLAUDE.md, MCP servers, and plan mode to create an agentic coding system that learns your codebase and automates workflows.
HG-RAG Beats Flat Retrieval on Graph Queries Across 800-Node Worlds
HG-RAG uses graph-traversal over knowledge graphs for RAG, beating flat retrieval on hierarchical and multi-hop queries across worlds up to 800 nodes.
Supermicro: Double-Wide Racks, Liquid Cooling, and the Storage Bottleneck
Supermicro CBO says double-wide racks and liquid cooling are standard for AMD Helios and NVIDIA Vera Rubin, and storage is now the main AI bottleneck.
Inside Shopify Hack Days: Building a prototype for music-playing pages (2026)
Shopify's 2026 Hack Days produced a prototype for music-playing product pages, involving 150 participants over 48 hours with a 200ms load time. This explores audio commerce for merchants.
UV_EXCLUDE_NEWER: The Environment Variable That Cuts Your uvx CI Time by
Set UV_EXCLUDE_NEWER with a date in your GitHub Actions cache key to avoid redownloading uvx tools each run, saving 40+ seconds of CI time per workflow execution.
OpenAI GPT-5.6 Sol matches Fable 5 at 1/3 cost, adds multi-agent API
OpenAI's GPT-5.6 Sol nearly matches Claude Fable 5 on aggregate benchmarks at one-third the cost, with new multi-agent and tool-calling APIs.
Build an Adversarial Verifier Loop in Claude Code: Catch Bugs Before They Land
Stop trusting Claude Code's self-reports. Add a 3-verifier panel that refutes changes with concrete repro cases, catching bugs tests miss. Capped at 3 rounds.
MCP Cuts Token Costs 75% But Adds 30x Latency vs REST APIs
MCP cuts token costs by 75% but adds 30x latency versus REST. The protocol, backed by Anthropic and OpenAI, trades speed for dynamic tool discovery.
Feature Freshness: The Production Bug That Makes Good Recommenders Look Bad
Jie Li's article reveals that stale features—outdated user signals—can degrade recommender performance by 20-30% in offline metrics, often misdiagnosed as model problems. The piece urges teams to prioritize feature freshness monitoring alongside model tuning.
GraphRAG Memory Design: Retrieval Over Storage, MCP Integration
Agent memory design prioritizes retrieval over storage, using unified MongoDB and MCP server. GraphRAG enables multi-hop traversal via three strategies.
7 Breaking Changes in the 2026-07-28 MCP Spec: Your Before/After Migration Guide
The 2026-07-28 MCP spec removes sessions and the initialize handshake. Run these 7 greps against your src/ to find every breaking change, then migrate in order: sessions first, then handshake, then error codes.
MCP Server Versioning: How to Avoid Breaking All Your AI Clients (Like I
Stop breaking AI clients with MCP schema changes. Use query param versioning (?v=2) — it works with every MCP client, requires no code changes, and lets old and new versions coexist seamlessly.
Building a Production-Ready Snowflake MCP Server: A Practical Guide
A technical guide details building a production-ready Snowflake MCP server with OAuth 2.0, schema filtering, and rate limiting for enterprise AI agents.
How Simon Willison Ported a 0.2B Image Model to the Browser with Claude
Simon Willison used Claude Code to port a 0.2B image inpainting model to WebGPU, running it as a parallel side project while his main agent worked on Datasette. The technique? Research with Claude.ai, then hand off to Claude Code with research.md.
Claude Code's June 15 Agentic Credit Split: How to Avoid Hitting the $20 Wall
Claude Code's June 15 agentic credit split moves `claude -p` and CI workflows to a separate $20/month bucket on Pro. Upgrade to Max 5x or switch to direct API for production pipelines.
Prism v1.8 Adds CLI, MCP Server, and SDKs — Here's How to Use Them with
Prism v1.8's MCP server gives Claude Code direct control over caches, budgets, and routing. Install it in 2 minutes and ditch the dashboard for terminal-based AI infrastructure management.
OpenAI's ChatGPT 'Dreaming' Memory Retains Preferences Across Sessions
OpenAI launched a dreaming memory system for ChatGPT that retains user preferences across conversations by compressing and replaying session data, enabling persistent personalization.