// benchmarking AI where it matters
AI models are ranked on questions. But we use them to build software.
The Problem
Most benchmarks test whether a model can answer questions. That's not the job. The job is building software that works.
Most benchmarks don't tell you:
- Which model builds a working app
- Which one is fastest
- Which one is cheapest
- Which one actually works across repeated runs
The Solution
BuildBench tests AI by making it build real software. Each model gets the same project, the same features to implement, the same tests to pass. No human help. Just the AI.
input: project spec + test suite
constraint: no human intervention
evaluation: automated test pass/fail
runs: 5 per task for consistency
What We Measure
Score, pass rate, and reliability are the primary signals — they say whether the software works. Cost and speed are secondary: useful, but only read alongside the score, never instead of it.
Score primary
How much of the task actually worked — graded 0–100 by automated tests. We report the mean and the median (they diverge), with the score band. This is the signal that matters most.
Pass Rate primary
Of every run, how many fully passed, partially passed, or failed. A high average score with a low pass rate means it rarely finishes the whole job.
Reliability primary
Every task runs five times. Consistency is the share of repeated runs that fully pass — whether the model does the same thing every time, not just once on a good day.
Cost & Speed secondary
Tokens, dollars, and wall-clock time per run. Directional only — comparable just when the task set, protocol, and run counts match — so we report them next to the score, not in place of it.
Why It Matters
You don't need the "smartest" AI. You need the one that finishes the job, costs less, and works reliably.
Each report shows:
- What the AI built
- What passed and what failed
- How much it cost
- How long it took
- Whether it does the same thing every time you run it
Score tells you if it works. Cost and time tell you if you can actually use it — and for engineers, that's not a footnote:
Cost decides where it fits
A model run thousands of times in CI, an agent loop, or per pull request multiplies fast. A 10x cheaper model that scores nearly the same is the one you can afford to put in the inner loop — the expensive one stays a one-off.
Time decides if it breaks flow
Iteration speed is the job. A model that lands a working change in seconds keeps you in flow; one that takes twenty minutes per task stalls code review, blocks the pipeline, and gets abandoned no matter how smart it is.
The Goal
Find the AI that actually builds useful software. BuildBench already measures more of real software delivery than most answer-style benchmarks. The roadmap prioritizes benchmark integrity before platform expansion.
Modular Runner
Separate protocol, orchestration, evaluation, scoring, and reporting into discrete modules.
Asset Boundaries
Public task assets help the model without leaking private scoring signals.
Auditability
Report replay, rerun validation, baseline comparison. Historical results stay trustworthy.
Task Coverage
Expand tasks per category. Category conclusions are weak when one task dominates the signal.
Category Prompting
Backend, repair, algorithm, and web tasks get protocol guidance that fits the work.
Provider Diversity
Multi-provider parallel execution. Comes after correctness, reproducibility, and interpretability.
Leaderboard
Ranked by overall score. Read mean alongside median — they diverge — and weigh the trust badge: a smoke-test run is not a published comparison. Click a column to re-sort.
| Model | Overall | Median | Pass % | Cost | Trust |
|---|---|---|---|---|---|
| gpt-5.4-mini algorithm-strong, web-weak, operationally-fragile, budget-limited | 81.82 | 100 | 69.7% | $2.55 | smoke-test |
| kimi-k2.6 algorithm-strong, devops-weak, operationally-fragile, budget-limited | 67.55 | 79.62 | 50.3% | $3.74 | smoke-test |
By category
The blended score hides the shape. A model can be strong in one category and unusable in another — color is the score band, the number is the category mean.
| Model | algorithm | backend | data-sql | devops | repair | web |
|---|---|---|---|---|---|---|
| gpt-5.4-mini | 94.78 | 86.64 | 91.14 | 82.2 | 71.45 | 56.91 |
| kimi-k2.6 | 94.6 | 85.93 | 65.76 | 21.6 | 81.22 | 39.98 |