What Changed — The Pattern, Not a Product
You're deep in a Claude Code session, churning through a complex refactor. The agent is on a roll, making edits, running tests. Then: HTTP 429. Rate limited. Your flow is dead. You wait. You retry. You lose context.
This isn't a Claude Code bug. It's an architectural vulnerability. The LLM waterfall pattern—a failover cascade from primary provider → aggregator → local model—turns that single point of failure into a resilient chain. The pattern itself is provider-agnostic, but it maps perfectly onto Claude Code's architecture.
What It Means For You — Concrete Impact on Daily Claude Code Usage
Claude Code talks to Anthropic's API. If you hit rate limits (common during heavy sessions or peak hours), your session stalls. With a waterfall:
- Tier 1 (Primary): Anthropic API. Best results, highest quality.
- Tier 2 (Aggregator): OpenRouter or Portkey. Routes to alternative models (e.g., Claude 3.5 Sonnet via another provider) with fresh quotas.
- Tier 3 (Local): Ollama running a smaller model like Llama 3 or Mistral. Always available, zero cost, never rate-limited.
The system detects the 429 and automatically retries the next tier—within milliseconds. Your Claude Code session doesn't break; it gracefully degrades.
Try It Now — Setting Up a Fallback for Claude Code
You can't change Claude Code's internal API calls directly, but you can wrap it in a proxy that implements the waterfall. Here's a minimal setup using Python with httpx:
# waterfall_proxy.py
import httpx
import os
ANTHROPIC_KEY = os.getenv("ANTHROPIC_API_KEY")
OPENROUTER_KEY = os.getenv("OPENROUTER_API_KEY")
OLLAMA_URL = "http://localhost:11434/api/generate"
tiers = [
{
"name": "anthropic",
"url": "https://api.anthropic.com/v1/messages",
"headers": {"x-api-key": ANTHROPIC_KEY, "anthropic-version": "2023-06-01"},
"model": "claude-sonnet-4-20250514"
},
{
"name": "openrouter",
"url": "https://openrouter.ai/api/v1/chat/completions",
"headers": {"Authorization": f"Bearer {OPENROUTER_KEY}"},
"model": "anthropic/claude-3.5-sonnet"
},
{
"name": "ollama",
"url": OLLAMA_URL,
"headers": {},
"model": "llama3"
}
]
def waterfall_request(prompt):
for tier in tiers:
try:
with httpx.Client(timeout=30) as client:
if tier["name"] == "ollama":
resp = client.post(tier["url"], json={"model": tier["model"], "prompt": prompt, "stream": False})
else:
resp = client.post(tier["url"], json={"model": tier["model"], "messages": [{"role": "user", "content": prompt}]}, headers=tier["headers"])
resp.raise_for_status()
return resp.json()
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
print(f"{tier['name']} rate limited, falling back...")
continue
raise
raise Exception("All tiers failed")
# Example usage
response = waterfall_request("Refactor this function to use async/await")
print(response)
Then point Claude Code to this proxy:
# In your terminal, before running claude:
export ANTHROPIC_BASE_URL="http://localhost:8000" # Your proxy
claude
Pro tip: For local fallback, ensure Ollama is running with ollama serve and a model downloaded (ollama pull llama3). The local model won't match Claude's quality, but it keeps your session alive during outages.
Beyond Failover: Cost and Latency Optimization
Use the waterfall for more than emergencies:
- Cost-aware routing: Put a cheaper model (e.g., Claude Haiku) in Tier 1 for simple tasks like summarization, with Sonnet as fallback for complex reasoning.
- Latency-first: If speed matters more than quality (e.g., autocomplete), put Ollama first, with cloud APIs as fallback for nuanced work.
The LLM waterfall pattern transforms Claude Code from a brittle single-point dependency into a self-healing system. Configure it once, and your sessions survive peak traffic, provider outages, and cost spikes—zero downtime, zero manual intervention.
Source: dev.to
