Head-to-head benchmark

Benzi vs Claude Code

Same bugs, same repos, same grading. Benzi — an index-grounded coding agent running a cheap model (DeepSeek v4-flash) — goes head-to-head with Claude Code on Sonnet across five real bug-fixes in five languages. It matches Sonnet on correctness for about a seventh of the cost.

Method. SWE-bench-style. Each task is a real merged bug-fix commit; the tests are kept, only the source is reverted, every agent is handed the same issue text, and grading is differential — the target test must go green and nothing else may regress, with a validity guard so a build failure never counts as a pass. Git history is hidden during the run, so no agent can read the answer. Both harnesses build & run the repo's own tests, worktree-isolated.

5 / 5

solved by the cheap model — full parity with Sonnet

≈7×

cheaper than Claude Code / Sonnet

$0.46

DeepSeek's cost for all five fixes (vs $3.53)

languages · Python · Java · C++ · C# · JS

The one place Sonnet wins: the clock The cheap model runs roughly 35% more wall-clock overall — it converges in more turns, then verifies each edit against the repo's own tests before calling it done. It lands within ~30% of Sonnet on Java, C++, and Python; the gap is a 49-second difference on C# (large only as a percentage of a tiny task) and stands out on JavaScript (~2×, and its most variable — a fast convergence lands within ~35%). That's the whole shape of it: more iterations for ~7× lower cost, at identical correctness — and it's a gap we're actively closing (faster build/verify loops, tighter iteration). An engineering dial, not a ceiling.

C++Solved

fmt

+26% wall · ~7× cheaper

Compile-time template metaprogramming — cracked by building the test to convergence. $0.22 vs $1.61.

JavaSolved

commons-cli

+7% wall · ~7× cheaper

The index drops onto the null-option path directly — matched clock. $0.04 vs $0.31.

PythonSolved

sqlglot

+19% wall · ~14× cheaper

A diffuse dialect-generation bug anchored to a SQL feature name; all three cases green. $0.07 vs $1.01.

C#Solved

CsvHelper

+69% wall · ~6× cheaper

A nullable-generics pattern-match trap — just 49 s slower on a tiny task. $0.05 vs $0.28.

JavaScriptSolved

dayjs

~2× wall · ~5× cheaper

A floating-point rounding bug — the run's one real clock gap. $0.07 vs $0.32.

C++

fmt — 'c' format out-of-range

github.com/fmtlib/fmt ↗

fmt::format("{:c}", 256) and negative values were silently mistranslated instead of rejected; the fix must raise "character value out of range" for anything outside [0, 255] while keeping in-range values formatting correctly. The fix threads through fmt's compile-time formatting machinery — deep template metaprogramming, where the error surfaces at instantiation, far from the edit.

The prompt (given verbatim to every agent)

fmt mishandles out-of-range integers formatted with the 'c' presentation type. `fmt::format("{:c}", 256)`, a negative value like `fmt::format("{:c}", -1)`, and other values outside a single character's range are silently mistranslated instead of being rejected. The 'c' type treats character values as unsigned, so the representable range is [0, 255]; anything outside it (negative, or > 255) should raise a clear "character value out of range" error, while in-range values (e.g. 200, 255) must still format to the corresponding character. Please fix it and keep the existing test suite passing.

Arm	Solved	Turns	Wall	Cost
BenziDeepSeek v4-flash	✓	77	541 s	$0.22
BenziSonnet	✓	68	401 s	$1.74
Claude CodeSonnet	✓	59	428 s	$1.61

→The hard one — solved by building to convergence. The fix lives inside fmt's template machinery, impossible to reason about blind: the error surfaces at instantiation in the test, not at the edit site. So each agent builds and runs format-test to convergence, reading real instantiation errors and fixing forward until green. DeepSeek runs more of those slow C++ build loops (77 turns, +26% wall) yet bills only $0.22 vs $1.61 — ~7.3× cheaper.

Why the build/test loop matters Static reading gets an agent to the region; it can't tell you whether a template-heavy edit actually compiles, because the error surfaces at instantiation in the test, not at the edit site. Giving the agent a compiler and the repo's own tests turns a blind guess into a feedback loop — the exact difference between failing and solving fmt. It's also why C++ eats wall-clock: that loop is only as fast as g++.

Java

commons-cli — null option NPE

github.com/apache/commons-cli ↗

Passing a null option name to CommandLine's lookup threw a NullPointerException instead of resolving to "no such option." getOptionValue((String) null, "default") should return the default; (null, null) should return null.

The prompt (given verbatim to every agent)

commons-cli throws a NullPointerException when an option name of null is passed to CommandLine's option lookup. For example `commandLine.getOptionValue((String) null, "default")` should return the supplied default ("default"), and `commandLine.getOptionValue((String) null, null)` should return null -- but instead a null option name blows up with an NPE while resolving the option. Make a null option name resolve to no option (returning null / the default) rather than throwing, and keep the existing test suite passing.

Arm	Solved	Turns	Wall	Cost
BenziDeepSeek v4-flash	✓	15	64 s	$0.04
BenziSonnet	✓	15	105 s	$0.66
Claude CodeSonnet	✓	12	60 s	$0.31

→Decisive cost win, matched clock. Everyone solves the NPE, but Benzi/DeepSeek does it for 4¢ — ~7.8× cheaper than Claude Code ($0.31) and within four seconds on the wall. The index puts it on the null-option resolution path immediately, then confirms by running the suite.

Python

sqlglot — ARRAY_CONCAT_AGG transpile

github.com/tobymao/sqlglot ↗

Transpiling BigQuery's ARRAY_CONCAT_AGG to DuckDB produced wrong SQL whenever the aggregate carried an inner ORDER BY or LIMIT — it silently mistranslated instead of round-tripping (and LIMIT, which DuckDB has no equivalent for, should be reported unsupported). The fix lives in dialect-generation logic spread across an inheritance hierarchy — the trap is finding it, not writing it.

The prompt (given verbatim to every agent)

Transpiling BigQuery's ARRAY_CONCAT_AGG to DuckDB produces wrong SQL when the aggregate has an ORDER BY or a LIMIT inside it. For example `SELECT ARRAY_CONCAT_AGG(arr ORDER BY y) FROM ...` and `SELECT ARRAY_CONCAT_AGG(arr LIMIT 2) FROM ...` do not round-trip correctly to the duckdb dialect. LIMIT inside ARRAY_CONCAT_AGG has no DuckDB equivalent and should be reported as unsupported rather than silently mistranslated. Please fix the transpilation and make sure the existing test suite still passes.

Arm	Solved	Turns	Wall	Cost
BenziDeepSeek v4-flash	3 / 3	56	371 s	$0.07
BenziSonnet	3 / 3	21	491 s	$0.42
Claude CodeSonnet	3 / 3	35	312 s	$1.01

→Both nail all three cases, and the clock is close. The bug is anchored to a SQL feature name, not a code symbol, so pinning it is part reading and part search — DeepSeek does more of both (56 turns) but stays within ~19% on wall-clock and lands every case for $0.07 vs $1.01.

CsvHelper — nullable generic converter

github.com/JoshClose/CsvHelper ↗

A custom converter deriving from the generic TypeConverter<T> base, where T is a nullable value type (say bool?), threw InvalidCastException when asked to convert a null to string — it should serialize to an empty string, not blow up. The trap is C# pattern-match semantics: value is T v is false for a null nullable, so the base takes the throwing branch.

The prompt (given verbatim to every agent)

A custom type converter that derives from CsvHelper's generic TypeConverter<T> base, where T is a nullable value type (for example a converter for bool?), throws System.InvalidCastException when it is asked to convert a null value to a string. Writing a null nullable value should serialize normally to an empty string, not blow up. Fix the generic converter so converting a null of a nullable type to string works, and keep the existing test suite passing.

Arm	Solved	Turns	Wall	Cost
BenziDeepSeek v4-flash	✓	20	121 s	$0.05
BenziSonnet	✓	14	96 s	$0.28
Claude CodeSonnet	✓	10	72 s	$0.28

→A subtle language-semantics bug, both solved. Fixing it means seeing why null is T fails when T is Nullable<…> and correcting the null-to-string path in the generic base. DeepSeek takes more turns (20 vs 10) — 49 seconds more on a 72-second task — but lands it for $0.05 vs $0.28.

JavaScript

dayjs — duration float rounding

github.com/iamkun/dayjs ↗

Duration formatting leaked floating-point noise into the sub-second field: dayjs.duration(-2812).toISOString() returned -PT2.8120000000000003S instead of -PT2.812S. The fix rounds the accumulated seconds to millisecond precision. A behavioral bug — the test asserts the output string, so any correct rounding passes.

The prompt (given verbatim to every agent)

dayjs's duration formatting produces sub-second values polluted with floating-point noise. For example dayjs.duration(-2812).toISOString() returns '-PT2.8120000000000003S' instead of '-PT2.812S', and dayjs.duration(3121632.27382247).toISOString() returns 'PT52M1.6320000000000001S' instead of 'PT52M1.632S'. The sub-second component of the ISO string is not rounded to millisecond precision. Fix the duration ISO serialization so sub-second values are rounded to at most millisecond precision, and keep the existing test suite passing.

Arm	Solved	Turns	Wall	Cost
BenziDeepSeek v4-flash	✓	25	204 s	$0.07
BenziSonnet	✓	19	117 s	$1.04
Claude CodeSonnet	✓	10	100 s	$0.32

→The run's one real clock gap — and still a solve for a fraction of the cost. Both trace the noise to how the seconds field accumulates a float and add the millisecond rounding, confirming against the duration tests. DeepSeek takes ~2× the wall-clock here — the task we're using to drive the iteration loop down — while still fixing it for $0.07 vs $0.32.