Token-Saving Tools Overpromise: Real Benchmark Shows 6–32% Savings, Not 60–90%

Key Takeaways

• Token-saving tools deliver 6–32% savings, not 60–90%.
• In Claude Code, lazy MCP loading means tools often go unused—enable them with hooks and measure full sessions.

What Changed — The Benchmark That Deflates Token-Saving Claims

A developer (who works on repowise, one of the tools tested) published a rigorous, preregistered benchmark of 5 token-saving tools across Codex and Claude Code. The setup: 48 Django questions from SWE-bench, 261 runs, same agent/prompt/repo/tool access, fresh index per tool, and a no-tools baseline. The results on Codex (gpt-5.6-sol) are sobering:

• repowise: -31.6% output tokens (p<0.0001)
• CodeGraph: -24.4% (p<0.0001)
• Serena: -14.8% (p<0.0001)
• Graphify: -8.9% (p=0.003)
• code-review-graph: -6.0% (p=0.046)

None came close to the 60–90% claims. Even the best result (31.6%) is half of the lowest typical claim. And when corrected for multiple testing, only three of the five reductions hold up statistically.

What It Means For You — Claude Code Users, Listen Up

The most important finding for Claude Code users: under Claude Code, most of these tools were barely called at all. code-review-graph was never called once across 15 questions. Graphify was called 3 times, Serena 4. Nothing changed on the tool side—same servers, same indexes, same questions. Codex called every tool on every question.

The likely culprit: Claude Code loads MCP schemas on demand, so the agent has to discover the tool before it can call it, and often never does. Codex mounts schemas up front, making tools immediately available.

This means that even if a token-saving tool could save 30% in theory, in practice it might save you 0% if Claude Code never invokes it. The benchmark author plans to rerun with hooks that enforce tool adoption—but until then, you should not assume any MCP-based token-saving tool is working.

Try It Now — What to Do in Your Claude Code Setup

1. Check if your MCP tools are actually being called. Run a session with verbose logging (claude --verbose) and grep for tool calls. If you see zero calls to your token-saving tool, it's dead weight.

2. Force tool usage with hooks. If you want to ensure a tool gets used, add a PreToolUse hook in your settings.json that injects a reminder to use the tool. For example:

   {
     "hooks": {
       "PreToolUse": [
         {
           "matcher": "Read",
           "hooks": [
             {
               "type": "command",
               "command": "echo 'Consider using your token-saving tool first.'"
             }
           ]
         }
       ]
     }
   }
   

   Or use a system prompt instruction in your CLAUDE.md: "Always use the codegraph_search tool before reading files."

3. Measure full sessions, not single payloads. The benchmark found that loading one commit's context through repowise takes 393 tokens vs. 13,984 for reading the changed files—a 35.6x compression. But across a full agent session, the savings drop to 31.6% on Codex and 15.9% on Claude Code. Agents re-read, backtrack, and re-plan, so single-payload numbers are misleading. Use claude --output-format json to capture token usage per session and compare with/without the tool.

4. Watch out for prompt-cache warming. The author nearly published a cost table showing code-review-graph as 43% cheaper, but that was an artifact of cache warming: the first arm paid full price, later arms reused cache. Always report output tokens, not cost, when comparing tools.

5. Indexing time is a real cost. repowise took 366.8s to index (with prose generation off; 1,058s with default init), while CodeGraph took 16.4s—22x faster. If you switch repos often, indexing time might eat your savings. Consider tools with faster indexing like CodeGraph (16.4s) or code-review-graph (44.8s).

Quality: No Winner, No Loser

A blind judge scored every tool (including repowise) slightly below the bare agent, but the differences (0.04–0.25 points on a 10-point scale) were smaller than the benchmark's own noise (0.69 points). So token savings don't come at a quality cost—but they don't help quality either.

Deterministic Retrieval: A Different Story

On a deterministic retrieval benchmark (ContextBench), repowise's get_answer found 87.6% of gold files from ~19 files served, while code-review-graph found 44.5% from 5.4 files (highest precision at 0.240). If you care about precision (fewer tokens), code-review-graph might be better despite lower coverage. But this benchmark used no LLM, so it doesn't reflect real agent behavior.

Bottom Line

Don't trust 60–90% claims. Real savings are 6–32%, and only if the tool actually gets called. In Claude Code, lazy MCP loading means many tools go unused—verify with hooks or verbose logging. And always measure full sessions, not single payloads.

Full methodology and raw data: github.com/repowise-dev/repowise/blob/main/docs/BENCHMARKS.md

Source: reddit.com