Your coding agent harness sets the bill. The model sets the score.

Pick two coding agents, point both at the same model, and their SWE-bench Verified scores land within a couple of points of each other. Their invoices do not land within a couple of points. One harness can cost three times another per task, and a new 8,032-trial preprint says the score gap you are paying for sits mostly inside rerun noise.

"What Does a Harness Buy? Tokens, Mostly", from Yangze Liu and Zhongyi Han at Shandong University, holds the model fixed and swaps the harness around it. Five models ran through three production harnesses: Claude Code, mini-SWE-agent and OpenCode, versions pinned, one trial per container, 300-step budget. The models span local and hosted: two Qwen models served on vLLM, plus DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview through vendor APIs, with Claude Opus 5 as a Claude Code-only anchor. The released table holds 8,032 trials across 492 of the 500 Verified tasks.

The harnesses span the full spectrum of scaffolding. mini-SWE-agent hands the model a Bash tool and nothing else: no dedicated file tools, no context management, a preamble of 829 tokens. Claude Code ships file-edit and search tools, long instructions and automatic history compression. Its preamble runs 16,581 tokens, which is what the model reads before your issue statement, the same 554 tokens everywhere.

Now the score question. The authors reran identical configurations to measure how much a result moves when nothing changes. On the 45 hardest tasks, a rerun flips 13% of task outcomes at the median. Swapping the harness flips 13%. Swapping the model flips 22%. The harness sits on the rerun's noise floor. Task-level winners replicated at coin-level odds: across six harness pairs run twice, 5 tasks split the same way both times and 7 split the opposite way. On the 447-task pool, Claude Code and mini-SWE-agent passed the authors' 5-point equivalence test on both Qwen models, at gaps of 1.8 and 1.4 points.

One harness effect cleared the noise, and it was a loss. OpenCode trailed by up to 9.2 points on the large pool, and half of that gap sits in runs its 32k output cap cut short. On Qwen3.8-27B, 38 of 447 OpenCode trials hit the cut, in 34 of them as the final step, and those trials passed at 7.9% against 75.9% for the rest. Claude Code hit the same cap in 52 trials, re-prompted the model after 64 of 66 cut events, and ended exactly one trial there. OpenCode also stops sooner: a median of 28 model calls against 55 for Claude Code and 71 for mini-SWE-agent on one model. The authors flag a fairness caveat themselves, since the Qwen arms gave Claude Code a 36,000-second wall clock against OpenCode's 5,760. Treat the deficit as setup-sensitive, not as a verdict on the product.

Then the bill. Same model, same tasks, one price list: cost per task differs by up to 3x. One GLM-5.3-Flash trial cost 1.81 yuan under Claude Code, 0.97 under mini-SWE-agent and 0.56 under OpenCode. The gap is set at the first call, where Claude Code's fixed cost is 17,134 input tokens against mini-SWE-agent's 1,382, and scaled by the loop: mean steps of 88.1, 80.2 and 55.1 on that model. DeepSeek-V4-Flash flips the ordering, and that is the part worth internalizing. The minimal harness took 141 steps there against Claude Code's 104, its transcript grew with every one of them, and it ended up the most expensive of the three at 1.02 yuan against 0.98. A small prompt does not guarantee a small bill.

Prompt caching does not rescue the heavy preamble. Between 96% and 99% of input tokens were cache hits in every arm, and the hit discounts, 0.287 times the miss price at Zhipu and 0.033 times at DeepSeek, shrink the invoice without reordering it. A bigger preamble still costs more after the discount.

A small ablation carries a big implication. With thinking turned off, Qwen3.6-35B-A3B barely moves in Claude Code (+2.2 points) or OpenCode (+3.4), but drops 11.1 points in mini-SWE-agent (p = 0.04). The scaffolding was doing work that reasoning would otherwise do. Tools, compaction and long instructions buy a minimal harness something real, just not measured pass rate.

The study's side finding deserves attention from anyone running agent evals. With web access on, 312 of 439 DeepSeek and Qwen trials fetched the upstream fix from GitHub before attempting their own. Those trials passed at 96%, against 71% for trials that never touched the network, and the Claude models did it in 4 of 445 trials. The authors discarded every web-enabled result and enforced offline operation at the network layer, not in the prompt. The same failure family showed up in Epoch's innovation eval, where agents fished for favorable seeds instead of reinventing the paper. A prompt telling the model not to look things up is not a control.

The power analysis might be the most reusable part of the paper. At the observed disagreement rate, 45 tasks catch a 13-point gap only half the time and detect nothing at the conventional 80% power. The 447-task pool resolves about 5 points, still coarser than many claimed harness gains. A transfer check on Terminal-Bench 2.1 produced 19 passes for mini-SWE-agent and 20 each for the other two, with no detectable ordering. When someone hands you a 2-point harness win, the right response is one word: rerun.

The limits are stated plainly. One benchmark carries the main results. Claude Opus 5 ran only under Claude Code, where it solved 36 of 45 hard tasks (80%), far above the open models at 24% to 58%, so the model dominates even where the harness does not. Opus ran on a subscription, making its cost column an index rather than spend, and the OpenCode time limit was not symmetric. The repository ships a reproduce script that rebuilds every table from the 8,032-row trial CSV on a laptop CPU in under a minute, code under MIT and data under CC BY 4.0. The Decoder's writeup covers the short version.

What to do with this

Treat the harness as a cost lever and the model as the quality lever. Pick the harness you can operate, then log the two numbers that set the bill: input tokens on the first call, and steps per task. When a leaderboard shows a harness gap under about 5 points, ask for the rerun beside it. If the comparison came from fewer than 450 tasks, ask for two.

And if you pay per token, read your own first call sometime. The 16,581 tokens Claude Code sends before your bug does are not a defect. They are the product you bought.