Databricks CTO Matei Zaharia's team benchmarked coding agents on the company's own polyglot codebase. The results show open-source GLM-5.2 matching top closed models and a minimal harness beating vendor harnesses at half the cost.
Key facts
- Pi harness matched vendor harnesses at 2x less cost.
- GLM-5.2 matched top closed models on Databricks' polyglot codebase.
- Sonnet 5 cost more per task than Opus 4.8 despite lower per-token price.
- Per-token rate cards are 'nearly useless' for agentic workloads.
- Omnigent open-sourced under Apache 2.0 as a 'meta-harness'.
Public coding benchmarks have a dirty secret: they all look suspiciously alike. SWE-bench and Terminal-Bench are dominated by Python-ish repos with fixed tests — and your codebase probably isn't. So when Databricks CTO Matei Zaharia's team wanted to know which coding agents actually work, they built an internal benchmark on a sample of their own codebase — a polyglot beast of Scala, Go, Rust, Java, TypeScript, Protobuf, Jsonnet, and more — and published what they found According to the source.
The findings are the most useful coding-agent data of the summer, because two of them cut directly against how most teams choose their stack. Zaharia was careful about the epistemics up front: these are results on their sample of their codebase, not meant to be comprehensive — but many companies can run a similar internal benchmark.
Key Takeaways
- Databricks benchmarked coding agents on its own polyglot codebase.
- GLM-5.2 matched top closed models, a minimal harness halved costs, and cheaper-per-token models cost more per task.
The top tier is crowded — and open source is in it
The first result is the least surprising but sets the stage: many models are now competitive at the top tier, including open source. The era when one closed model was unambiguously ahead on real engineering work is over; on Databricks' internal tasks, multiple models cluster at the top. That has a practical consequence: if several models are near-equivalent on your tasks, then model choice stops being the decision that matters most — and the decisions that do matter shift downstream, to the harness and the economics.
GLM-5.2 holds up far from benchmark-land
The sharper version of finding one: GLM-5.2 in particular was a major step forward in open-source coding-agent performance — even on a codebase that looks nothing like SWE-Bench or TerminalBench. The standing objection to open-model benchmark wins has always been distribution overlap: maybe the model is great at the kind of Python-repo task benchmarks measure, and falls apart on your Scala services and Protobuf schemas. Databricks' codebase is close to a worst-case test of that objection — heavy in Scala, Go, Rust, Java, TypeScript, Protobuf, Jsonnet — and GLM-5.2 delivered anyway. This independent, adversarially-different data point confirms that GLM-5.2's FrontierSWE numbers survive contact with messy, unfamiliar reality. Per the arXiv preprint on MLA, the price reversal phenomenon formalizes why cost-per-token metrics mislead for agentic workloads.
The harness result nobody should skip
Harnesses make a huge difference in cost-performance. Specifically — the very simple Pi harness (Mario Zechner's minimal, four-tool agent) achieved the same success rate as the LLM vendors' own harnesses running Opus and GPT-5.5 — at 2x less cost. The mechanism, per Zaharia: mainly smaller inputs to the LLM. Every token of scaffolding a harness injects — tool descriptions, boilerplate instructions, machinery — is a cost you pay on every model call of every step. Pi's minimalism turns out to be economics.
The price-tag trap: cheaper per-token ≠ cheaper per-task
The subtlest finding is the one most likely to be costing you money right now: cheaper per-token does not imply cheaper per-task. In Databricks' runs, Sonnet 5 costs less per token than Opus 4.8 — but used more tokens, resulting in higher total cost and lower quality. The cheaper model took more steps, more retries, more verbose reasoning to do the same work — and lost on both axes. Zaharia points to research from his former student Lingjiao Chen formalizing it as the "Price Reversal Phenomenon" — the first systematic study showing listed API prices routinely misrepresent actual inference costs for reasoning models. The company's blog post provides the full write-up on how the benchmark was built.
What Databricks is doing about it
The findings explain two of the company's recent bets. Omnigent — the open-source (Apache 2.0) "meta-harness," a harness of harnesses in Zaharia's phrase — exists because you want to switch and compose agents and harnesses per task rather than marry one. And Unity AI Gateway exists because someone needs to analyze and gate LLM usage centrally, since costs hide in places rate cards don't show. But the most transferable takeaway isn't a product — it's the method: sample your own codebase, run the candidates through it, and measure success rate and cost-per-task yourself.
What to watch
Watch for adoption of Omnigent by other enterprises running internal agent benchmarks, and whether the 'Price Reversal' paper influences how major cloud providers price agentic workloads. The Q3 enterprise spending reports from Databricks and Snowflake will reveal whether cost-per-task metrics replace per-token pricing in procurement decisions.
Source: pub.towardsai.net
