跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.04433 · v2

What Does a Harness Buy? Tokens, Mostly

首次提交 2026-10-03 18:47
版本更新 2026-11-18 21:24
雷达观测 2026-11-19 15:04
arXiv ↗PDF ↗

摘要

A coding agent is a language model inside a harness: the system prompt, the tools, and the context management that let a chat model work in a repository. Most of the engineering on these agents goes into the harness, and vendors and leaderboards credit it with score gains. Earlier studies have fixed the model and swapped the harness, but have not measured whether the swap changes the pass rate more than running the same harness again. We run seven models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified and two benchmarks of other kinds, and rerun them unchanged. No harness leads consistently across models. Where the data can bound the difference between two harnesses, the bound is 2 to 5 points, and a harness swap changes about as many task outcomes as a rerun. The exceptions are harness effects in particular pairings of model and harness, whose direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. Cost differences between harnesses also depend on the model, from under 10% to 3 to 4 times. Which harness is better therefore has no answer apart from the model, and a harness difference should be reported next to a rerun.

作者备注:31 pages, 9 figures. v2: seven models, reruns of every model, two non-SWE benchmarks; revised conclusions on pass rate and cost. Code and data: https://github.com/YangzeLiu/what-does-a-harness-buy

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 21:24 · 最新观测版本

What Does a Harness Buy? Tokens, Mostly

A coding agent is a language model inside a harness: the system prompt, the tools, and the context management that let a chat model work in a repository. Most of the engineering on these agents goes into the harness, and vendors and leaderboards credit it with score gains. Earlier studies have fixed the model and swapped the harness, but have not measured whether the swap changes the pass rate more than running the same harness again. We run seven models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified and two benchmarks of other kinds, and rerun them unchanged. No harness leads consistently across models. Where the data can bound the difference between two harnesses, the bound is 2 to 5 points, and a harness swap changes about as many task outcomes as a rerun. The exceptions are harness effects in particular pairings of model and harness, whose direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. Cost differences between harnesses also depend on the model, from under 10% to 3 to 4 times. Which harness is better therefore has no answer apart from the model, and a harness difference should be reported next to a rerun.

查看此版本解读 →查看此版本原文 ↗