Key Takeaways
- Ora's benchmark shows Vercel's eve beats Claude Code with 7% fewer steps and 2x native success on live sites.
- Claude Code users should evaluate eve for web-integration tasks.
What Changed — Ora's Live-Site Benchmark Reveals How Claude Code Stacks Up
Ora, a platform that benchmarks AI agents on live websites, ran Claude Code against Vercel's eve framework across hundreds of real user journeys. The results, published on Vercel's blog, give Claude Code users a rare look at how their tool performs in real-world web tasks—not just coding.
Ora's agents attempt to sign up for products, integrate with them, and pay. They fail often: Ora estimates 99% of the web isn't agent-ready. But the benchmark shows which harnesses handle the chaos better.
The numbers:
- 7% fewer steps for eve to reach the goal
- 2x native success — eve finished twice as many tasks on the customer's own site instead of falling back to web search
- 9% more valid endpoints — more of the endpoints eve found were actually callable
Both harnesses ran the same models: Claude Fable 5 and Haiku 4.5. The only variable was the harness—the software that gives the model tools and drives it step by step.
What It Means For You — Claude Code vs. eve in the Real World
Claude Code is a coding agent. It excels at repo-level tasks: editing files, running tests, committing code. But Ora's benchmark tests a different kind of task: navigating live websites, filling forms, and making API calls. That's where eve, built on Next.js, shines.
For Claude Code users, this doesn't mean your tool is obsolete. It means:
Know your use case. If you're writing code, Claude Code is still your best bet. If you're building an agent that interacts with the web, consider eve.
Harness matters more than model. The same model (Claude Fable 5) performed differently under different harnesses. The harness's tool design and step-driving logic have a huge impact.
Native success is a differentiator. eve's 2x native success means it uses the site's actual UI and endpoints more often, rather than searching the web for alternatives. That's critical for tasks like "integrate with this product."
Try It Now — How to Apply This to Your Workflow
1. Evaluate your agent's harness
If you're building agents that interact with websites, don't assume Claude Code is the right harness. Run your own mini-benchmark:
# Test Claude Code on a live site task
claude code "Sign up for a trial at example.com, integrate the API, and make a test request. Report every step."
Then try the same with eve (if you're on Vercel) or another harness. Compare steps, success, and cost.
2. Optimize for native success
If your agent falls back to web search too often, it's a sign the harness isn't using the site's structure well. Look at:
- Endpoint discovery: Does the harness find and call valid endpoints directly?
- UI interaction: Can it fill forms and click buttons without search?
- Prompt caching: Ora found a prompt-caching issue in eve that led to a 15% cost reduction after fixing. Check your own agent's caching.
3. Use Ora's journey tool
Go to journey.ora.ai and run a journey on your own site. You'll see cost, latency, and steps for each agent. That's the same data Ora used to make this comparison.
Conclusion
Claude Code remains a powerhouse for coding, but this benchmark is a wake-up call: for web-facing agent tasks, the harness matters more than the model. If you're building agents that need to sign up, integrate, and pay on live sites, eve's numbers are worth a look. But if you're shipping code, Claude Code is still your workhorse—just know its limits.
Source: vercel.com
[Updated 23 Aug via vercel_blog]
Ora's co-founder Assaf Elovic previously built Tavily, a web search engine for AI agents acquired by Nebius earlier this year, and co-founded Ora with Liad Yosef to measure web agent-readiness. The platform runs on Vercel, with the agent runtime on the same infrastructure. Ora's engineering lead, Ido Finder, noted that side-by-side coverage of every harness is a key value. After benchmarking eve, Ora now builds on eve itself, citing the sandbox override as a decisive feature. The 16-person team ships hundreds of commits daily, with coding agents handling infrastructure work.
