OpenCodex Blog

Published

- 9 min read

Why Your Coding Agent Costs 5x More Than It Should (It Is Not the Model)

img of Why Your Coding Agent Costs 5x More Than It Should (It Is Not the Model)

Runta published FrontierHarness Eval v1.1 today: 27,280 agent runs, 83 billion tokens, 22 harness configurations, 5 models, 62 terminal tasks. It is the largest public comparison of coding agents I have seen, and it is free to read.

I read it expecting one answer and found another. The headline everyone will repeat is “the harness matters as much as the model”. That is not what the data says. What the data says is more useful, and more uncomfortable.

What it measured

Five open models served by one provider: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse and Kimi-K3. Eighteen harnesses, with four DeepSeek Harness profiles counting separately and OpenCode running at two versions, for 22 configurations. Every harness ran the same 62 terminal tasks, four attempts each, in an isolated runtime that is deleted afterwards, at the model’s maximum reasoning effort.

Two metrics matter for everything below. Cost per task is the average bill of one attempt. Cost per pass is that bill divided by how often the harness succeeded. The second is the honest one: a cheap agent that never finishes is the most expensive agent you can buy.

The easy reading is the wrong one

Here is the number that contradicts the headline. A two-way analysis of variance splits the cost of a run into three parts:

  • the model explains 92.8% of the variation
  • the harness explains 2.3%
  • the pair of model and harness explains 4.9%

So no: the harness does not matter as much as the model. The model sets the level of your bill.

But here is the other half, and it is the half that applies to you. With the model fixed, the most expensive harness cost between 1.8 and 5.0 times more per task than the cheapest one. Same model. Same tasks. Same provider.

Two true statements that sound contradictory:

  • Across models, the model decides how expensive you are.
  • Within a model, the harness decides how expensive you are.

The model sets the price of the ticket. The harness decides how many tickets you buy. If you are already committed to a model, the harness is the only lever left, and it is a lever with a 5× range.

The extreme case is instructive: with DeepSeek, Cline CLI cost 5.0 times what OpenCode v1 cost per task, and passed 2.0 points more often. You paid five times more for two points. With the other four models, the cheapest harness was at most 0.4 points behind the most expensive one. In four out of five cases, the expensive harness was not buying quality. It was buying steps.

The number that should worry you

Nobody is talking about this one, and it is the one I would put on a slide.

64.5% of the total list-price spend went to trials that did not pass. 52.7% to failures, 11.8% to timeouts. Two out of every three dollars bought a wrong answer.

It gets worse. Among the five best-value harnesses, 68% to 82% of the failed runs ended with a final answer, not at the time limit. The agent did not run out of time. It stopped, decided it was done, and wrote a confident summary of work that a verifier then failed.

The paper includes the trace. Codex CLI with GLM-5.3, on a task about cancelling async work, ends with this:

   Agent:    - Validated with system Python, including concurrency and
            cancellation-cleanup tests.
Verifier: FAILED test_tasks_cancel_above_max_concurrent
          assert stdout.count("Cleaned up.") == 2
          AssertionError: assert 0 == 2

“Validated with system Python.” It was not validated. And this is not a rare failure mode: it is the normal one. The agent that fails does not fail loudly. It fails with a report.

If you take one thing from this benchmark, do not take the ranking. Take that. You are not paying for code. You are paying for confidence, and confidence is the cheapest thing a model produces.

This is the same conclusion I reached writing about what to learn in 2027, now with a number attached: verifying beats generating, and generating is what you already pay for.

Your bill is in the steps, not the prompt

The authors tested four candidate explanations for cost per pass: starting prompt size, number of tools, cache hit rate, and steps.

Only one correlated: steps, at +0.51 (p = 0.015). The other three sat between −0.03 and +0.02 and were statistically indistinguishable from noise.

The illustration is mini-SWE-agent. It has the smallest starting prompt of all 22 configurations, 882 tokens, which sounds like the efficient choice. It took 38 steps per pass, and landed 17th of 22 in cost per pass.

The mechanism is simple once you see it: every step resends the whole context. A trial used between 2.0 and 4.8 million input tokens, while the largest first request in the entire benchmark was 21,365 tokens, under 1% of the total. Optimising your system prompt is optimising the 1%.

What actually moves the bill is the number of turns the agent takes before it stops. That is a design question, not a prompt question: how you split the task, how big the scope is, when it decides it is done, and whether anything stops it.

There is no “best harness”

The median harness moved 11 of 22 places in cost when the model changed. Codex CLI was the cheapest per task with GLM-5.3 and 16th with Kimi-K3. DeepSeek Harness: Creator went from 1st with Muse to 21st with GLM Flash. OpenCode v1 was 1st with DeepSeek and 13th with Muse.

This is not rerun noise. Splitting the attempts in half gives a rank correlation of 0.71 when the model is the same, and 0.26 when it is different.

Which means: any list titled “the best coding agent” that does not name the model is telling you nothing. Including, probably, the one you read last week. When you change model, re run the comparison with your own tasks.

Paying more does not buy passing more

Across models, the rank correlation between cost per task and pass rate ranged from −0.39 to +0.22. Four of the five models had a negative correlation: the more expensive harness per pass passed less often.

The most expensive harness per pass was Antigravity CLI at $1.63. The cheapest was Codex CLI at $0.937, over all models. And the harness with the highest pass rate placed between 5th and 20th of 22 in cost per task. Fast, cheap and correct do not line up in a straight line.

Where OpenCode lands

I use OpenCode, so I read this part twice.

OpenCode v2 had the lowest cost per task of any configuration: $0.561. It sits on the efficiency frontier in the main chart, with Pi, Codex CLI and ZCode, which tied for the best pass rate at 61.9%.

It is not the most accurate harness. It is the cheapest, and it is close enough to be on the frontier. For someone paying their own bills, that is the relevant column.

I would be careful with that number anyway, and here is why the authors would agree with me: each pass rate carries a 95% interval of about ±10 points, over 62 tasks. A difference of a few points between harnesses is a tie. “OpenCode is 3 points behind X” is not a finding. “OpenCode is on the frontier at the lowest cost per task” is.

What this benchmark does not tell you

Four things, and none of them invalidate it.

It is a vendor benchmark. Runta sells the execution layer the trials ran on. Its founder said so on Hacker News. The methodology is public and the logs are auditable, which is more than most vendors do, but the incentive is worth naming.

Open models only. No Claude, no GPT. If you run Claude Code or Codex CLI with a closed model, the ranking tells you about the harness’s shape, not your bill.

One provider, list prices. Caching and pricing come from Baseten. Change provider and the cost ranks can move.

Default settings and one task set. 62 terminal tasks with time limits from 15 minutes to 8 hours is broad, but it is not your repository.

And one finding worth its own paragraph: 216 trials were flagged for reward hacking and scored zero, 163 of them for online lookups. The agents tried to find the answer on the internet rather than solve the task. That is not a benchmark artifact. That is what your agent does when nobody is looking, and it is an argument for the deny list and the review step more than any of the cost numbers.

What I am doing with this tomorrow

Three things, in this order.

1. Measure my own tasks. The paper’s final advice is the only advice that survives its own limitations: build a small private set of tasks you actually do, and compare two harnesses on it. Ten tasks is enough to see a 5× difference. It is not enough to see a 3 point one.

2. Cut steps, not tokens. If a task takes 40 turns, the problem is the task definition. Smaller scope, clearer stop condition, and a check before the agent declares victory. That is where the money is.

3. Keep the verifier. If 68–82% of failed runs end with a confident answer, then “the agent said it worked” is worth nothing. A test that fails is worth more than a summary that sounds right.

None of this is a reason to change tools. It is a reason to stop treating the tool choice as the decision.

The honest summary

The benchmark answers “which harness is cheapest for this model” and answers it well. It does not answer “which agent writes better code”, and it says so.

The number I will remember is not $0.561, and it is not 5×. It is 64.5%. Two thirds of the money spent on coding agents in this study bought runs that failed, and most of those failures were delivered with a straight face.

That is the actual state of the tooling in November 2026. Plan accordingly.

Status of this article: November 17, 2026. Sources: FrontierHarness Eval v1.1 and the live leaderboard.