Paper: coding harnesses mostly buy tokens—cost up to ~3×, pass rates close
arXiv:2610.04433 by Yangze Liu and Zhongyi Han (Shandong University) compares Claude Code, mini-SWE-agent, and OpenCode with the model fixed on SWE-bench Verified: heaviest vs lightest within ~5 points; OpenCode can trail by up to ~9; harness swaps and same-harness reruns both flip ~13% of tasks; cost per task can differ by up to ~3× from preamble tokens.
“What Does a Harness Buy? Tokens, Mostly.” (arXiv:2610.04433; Yangze Liu and corresponding author Zhongyi Han, Shandong University; arXiv v1 listed Sat 3 Oct 2026) treats a coding agent as a fixed backbone plus a harness—system prompt, tools, and context management. The authors run five models through Claude Code, mini-SWE-agent, and OpenCode on SWE-bench Verified and rerun cells to calibrate noise. The title answer: the clear purchase is the token bill, not a stable pass-rate lift.
On the ~447-task pool and the two models run there, Claude Code and mini-SWE-agent—the heaviest and lightest—land within about five points and are called equivalent. The one harness effect that clears noise is a loss: OpenCode can trail by up to about nine points on the large pool; on one model, roughly half that gap sits in runs cut short by an output cap—Claude Code prompts continuation, OpenCode often ends the step. On a 45-task hard subset across five models, swapping harnesses flips a median ~13% of task outcomes—the same rate as rerunning the same harness—while swapping models reaches ~22%. Wins do not stably attach to the same tasks across runs.
On cost, with the same model, tasks, and price list, cost per task differs by up to about 3×. The gap is set by the preamble of system prompt and tool schemas resent every step, scaled by step count; per-step growth and per-call tool output differ less. THE DECODER’s 2026-10-24 write-up restates these figures and notes that many advertised few-point “gains” sit inside rerun noise and limited statistical resolution. The authors urge publishing a same-setup rerun with every harness comparison. All numbers are taken from the paper/report text only.