Benzi on SWE-bench Verified

391 of 500 resolved (78.2%), pass@1, on DeepSeek v4-flash, for $37.33 total — 9.5¢ per resolved instance.

One configuration, one attempt per instance, no ensembling, no test-time selection, graded by the official SWE-bench harness. This report describes the system, the setup, the results, and how to reproduce them.

1. Headline results

Grading: swebench.harness.run_evaluation (swebench 5.0.2), official per-instance Docker images, official Hugging Face dataset. 499 generated patches were graded; one instance (django__django-13513) produced no patch and counts as unresolved without grading.

By repository

Every instance's turns, lines read, wall time, tokens, and cost are published row-by-row at benzi.fly.dev/benchmark_swebench.

2. The system

Benzi is a code-index agent: before the model sees anything, a tree-sitter-based compiler parses the entire repository and resolves it into a queryable map — symbols, call edges (with the evidence that resolved them), references, inheritance, per-scope data and control flow. Ambiguous calls keep their candidate lists instead of a guess; calls the compiler cannot classify are tagged as declared unknowns. The agent then works through ~36 structured tools over that index (profile, get_callers, backflow, trace_path, skim_source, …), each an O(1) lookup rather than a search. Writes are syntax-gated against the real language parser (a broken parse auto-reverts) and every landed edit reports its blast radius: the changed symbol, its callers, its holders.

The benchmark configuration adds three thin, deterministic layers around that core — all running on the same v4-flash model, all fail-open:

• Symptom map. Before the worker starts, the issue text is run against the index deterministically: traceback frames (weight 3), verbatim quoted code and strings grepped exactly (weight 2), and named symbols (weight 1) are resolved to concrete code sites and ranked by convergence. The ranked sites are appended to the task as evidence — the model still decides everything; it just starts from resolution instead of re-deriving it with fifteen turns of shell grep. When nothing resolves, the note says exactly that, which is itself signal.
• Scope cards. At the first edit of a turn, the agent is handed the compiler's profile of the scope it is editing; at commit time, profiles of every scope it touched. Localization context at exactly the moment of writing, from the same index.
• Verifier. After the worker finishes, a second v4-flash pass reads the issue and the final diff and returns agree-or-revise with a critique; on revise, the worker gets one capped correction round. Same model, sequential, fail-open — this is a second reading, not an ensemble; in practice the verifier requests a revision on nearly every instance, so its effect is a critiqued second draft rather than selective intervention.

No FAIL_TO_PASS or PASS_TO_PASS test names, files, or contents were ever shown to the agent at solve time. The agent sees the issue text and the repository — nothing else.

3. Setup

• Containers. Each instance ran inside its official SWE-bench per-instance Docker image, with the Benzi engine and a prebuilt Python runtime mounted read-only. Network access to GitHub and PyPI was blocked inside every container (hosts resolved to 127.0.0.1), so nothing could look up the fixed version of itself.
• Budgets. No wall-clock timeout was imposed on the agent.
• Retries. Instances whose attempt died to infrastructure (Docker image-pull failures on a full disk, API-layer errors, unparseable transport output) were re-attempted; the reported result is each instance's final attempt, and every crash remains recorded in the run ledger. No instance was retried for being wrong — only for never having run.
• Cost accounting. DeepSeek list prices over measured tokens. The 97% prompt-cache hit rate is what makes a 500-instance run cost less than forty dollars; the median resolved instance costs about a nickel and a half of generation.

4. Efficiency

The index is the efficiency story. A median instance is solved after reading 379 lines of source — not the repository, not a retrieval dump; the 27-turn median is mostly map queries whose answers are a few hundred tokens each. In our separate 24-bug cross-harness benchmark (benzi.fly.dev/benchmark), where reading is measured identically across harnesses, Benzi read 9,125–16,407 lines total (Sonnet / DeepSeek worker) against 20,704 for Claude Code and 43,598 for a plain DeepSeek harness on the same bugs — and the gap widens with difficulty. Lines-read is the one metric we compare across harnesses, because it is the one metric whose measurement we can make identical; wall-clock and dollar comparisons across differently-hosted models are published but not claimed.

5. Limitations

• Single configuration, single run, pass@1 — no ensembling, no multiple samples, no test-time selection, and no cherry-picking: every one of the 500 instances is reported from its one attempt.
• The verifier is the same model as the worker: it catches misreadings, not capability gaps.
• All numbers are on v4-flash; the architecture is model-agnostic but the result is not a claim about other models.
• The 24-bug efficiency comparison is in-house; its cross-harness fairness holds for lines-read only.

6. Reproducibility

The harness entry points are benzi_headless.py (the agent, one process per instance) and benchmark/swebench_docker.py (container orchestration, mounts, env). The benchmark configuration is env-flag complete: BENZI_VERIFIER=1, BENZI_VERIFIER_MODEL=deepseek, symptom map and scope cards on by default (BENZI_NO_SYMPTOM_MAP / BENZI_NO_SCOPE_CARDS unset). Generation ran on a single EC2 host (docker 25, python 3.12, swebench 5.0.2); grading used the official harness against the official dataset. The full per-instance ledger — including crashed attempts — is public at benzi.fly.dev/benchmark_full.

Benzi — an AI coding agent that doesn't read, it queries. benzi.fly.dev · github.com/oooscoos/Benzi