Same model. Four harnesses.

OpenCode is not in the race

It solved 6 of 24. On thirteen more it was still going when we stopped it, having already spent between 2.4× and 14.2× what Benzi needed to finish the same bug.

Those numbers are its best showing — measured only on the six bugs it actually fixed. Four more it finished with the wrong answer. Thirteen it never finished at all. Nothing here is an average dragged down by the failures; they are excluded entirely.

Benzi's KPI (Key Performance Indicator) — source lines read

Every harness opens more source as bugs get harder. The question is the slope. Each point is one bug; the 24 are laid out easiest to hardest, left to right.

Difficulty is Claude Code's turn count on that bug — a third-party yardstick, so no harness sets its own position on the axis. Lines read counts only what came back from file-read calls; grep and shell output are search, not reading. The figure beside each line is its slope: how many extra lines that harness opens per step of difficulty. Hover any point for the bug and its count.

Lines read counts only what came back from file-read calls; grep and shell output are search, not reading. The green figure in each row is the lowest of the four.

The same 24 bugs in the same order, with wall clock in place of lines read.

Wall clock is raw here — unlike the tables above, Benzi's per-repo index build is not subtracted, so these seconds run slightly higher than the warm figures quoted elsewhere on this page. Each point is that harness's most recent solved run for that bug; unsolved and unfinished runs are left out rather than plotted as fast. Benzi on Sonnet never solved http-parser and the DeepSeek harness never ran nats-server, so those two points are absent and neither enters its fit.

And the same again with dollars on the vertical axis.

Priced at the published per-token rates, same run selection as the chart above it. The two DeepSeek series run an order of magnitude cheaper than the two Sonnet ones, so at this scale they sit close to the baseline — the per-bug figures behind them are in the DeepSeek table further down. What the axis does show is the slope: Claude Code's cost climbs with difficulty faster than any other series here.

Priced at published per-token rates. The green figure in each row is the lowest of the four; the two DeepSeek columns are cheaper largely because that model costs roughly twenty times less per token. Blank cells are the two runs that never produced a fix.

Benzi vs Claude Code

This is the real comparison — two harnesses that both finish the job, on the same model. Benzi solves the same bugs in less time, for less money, having read less than half as much of the codebase.

Reading less is the mechanism, not a side effect. Claude Code opens 99 lines per read and takes big grabs; Benzi opens 43, because Benzi's Code Intelligence already knows which lines matter. Across the 22 timed bugs that is 16,851 lines against 7,525 — from almost exactly the same number of reads (171 against 174). Benzi does not look in fewer places. It looks in the same number of places and opens less of each one.

Every bug, both arms

Wall-clock and model calls per bug. Green means Benzi was faster.

Two bugs carry no timing, for opposite reasons. http-parser resists both harnesses: across every run recorded, Claude Code fixes it 1 time in 4 and Benzi 0 in 4 on this model, so a single result there measures luck rather than speed — it is counted as a loss for Benzi. nats-server is the reverse: Benzi has solved it 4 times in 5 attempts and Claude Code 2 in 4, and on this run Benzi fixed it while Claude Code did not. A stopwatch cannot compare a fix against a run that never produced one, so it is reported as a solve rate and kept out of the average — which costs Benzi the win rather than banking it.

Against DeepSeek's own harness

DeepSeek shipped its own agent harness on 13 August 2026. It defaults to deepseek-v4-flash — the same model Benzi runs here — so the model is held constant and the harness is the only thing that differs. Run interleaved, one bug at a time, both arms back to back.

This is the closest race on the page, and the honest summary is a tie on the thing that matters: both harnesses fixed all 23. Six percent on wall-clock is inside the noise — Benzi wins 11 bugs and loses 12. What is not close is how much source each opened to get there: 41,213 lines against 14,258, and DeepSeek Harness read more on every single bug. A 24th bug, nats-server, is excluded because neither arm produced a result — Benzi failed it, and the harness run died in cleanup before it could be graded. Benzi's wall-clock here is warm; the per-run index build is subtracted, though at 168s across 23 bugs it barely moves the number (cold, Benzi is still 4% ahead). DeepSeek Harness was v0.1.0-rc.6, released the day these ran.

Benzi DeepSeek vs OpenCode DeepSeek

On DeepSeek v4-flash — a model roughly twenty times cheaper — OpenCode holds up far better than it does on Sonnet: it finishes nine of these ten instead of six of twenty-four. It still ships a fix that fails the tests, and it is still slower, still more model calls, still more expensive, still reads more of the codebase to get there.

The reading gap is even wider here: across the nine both harnesses solved, OpenCode pulled 14,617 lines of source and Benzi 2,262. Same model, same bugs — one of them reads the codebase to find the answer, the other looks it up.

Totals cover the nine bugs OpenCode actually fixed. On the tenth, money, it stopped after 93 seconds with the bug still there — averaging that in would credit it for a fast give-up and understate Benzi (2.5× instead of 2.8×). The same rule removes nats-server from the Sonnet table above, where Claude Code failed and Benzi did not, and it is why http-parser counts as a loss for Benzi: a run that never fixed the bug cannot tell you how long fixing it takes.

Why that counts as a loss. money is graded against OpenCode even though it was the quickest run in the row, because the suite still fails afterwards and this benchmark settles order on correctness first, with speed only as the tiebreak. The same ordering is why OpenCode's slow runs are not left to run out: once one passes twice the slower of the other two arms on the same bug it is stopped and recorded as unfinished — not a failure, but not a completion either, and kept out of every average. A harness that is behind on accuracy and on time, model calls and cost at once has already answered the question the run was asking.

The tasks, with their prompts

The exact text handed to every agent, unedited.