- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks how much the harness (system prompt, tools, context management) changes a coding agent's results when the model is fixed. It runs five models through Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified, with reruns to measure run-to-run noise. Its conclusions are that harness swaps move pass rate about as much as reruns do, that the one clear effect is OpenCode losing tasks, and that harnesses mainly change cost, by up to 3×, driven by first-call overhead, context growth, and step count.
Critical strengths
- An important, practical question, and the right design for it: same model, same tasks, reruns as a yardstick, and paired tests.
- Careful, honest engineering: pinned versions, offline runs (after an audit found 71% of early web-enabled DeepSeek and Qwen trials fetched the upstream fix), and failure modes traced in the trajectories.
- The cost decomposition (preamble × steps, with growth per step) is clear and useful, and the power analysis shows readers what a 45-task comparison can and cannot resolve.
Critical weaknesses
Main concerns
-
The headline "not pass rate" is mostly a lack of power, not evidence of no effect. Equivalence is shown only for Claude Code vs. mini-SWE-agent, on the two locally served Qwen models, at a ±5-point margin wider than the gains vendors advertise. The other three models are tested only on the 45 hard tasks, where no gap is well powered, yet some point estimates there are as large as the OpenCode gap the paper does call real. The noise floor is also thin: two of the models have no reruns, and the task-level replication test rests on very few tasks. Matching flip rates (13% for reruns and harness swaps) show similar churn, not the absence of a consistent effect. "Not shown to differ" is supported; "barely moves" is not.
-
The paper's own significant results are harness effects on pass rate. OpenCode underperforms by about 8 to 9 points on Qwen3.8-27B, and both Qwen3.8 gaps survive Holm correction, which undercuts the broad framing even if the paper presents OpenCode as an exception. Removing limit-ended trials roughly halves the gap (Table A7), but limit-ended trials are themselves part of the harness outcome, and the recovery experiment did not establish the proposed mechanism (Appendix H).
-
Some comparisons are unequal, and the scope is narrower than the title. On the locally served Qwen arms OpenCode gets 5,760 s of wall clock versus 36,000 s for Claude Code; dropping the affected tasks does not shrink its gap (Section 4.3), but it weakens the design. More importantly for scope, for every non-Anthropic model, Claude Code runs through a compatibility layer (without its Grep tool), and Opus 5 is never run in another harness, so no Anthropic frontier model is compared across harnesses. The DeepSeek home-harness check is relevant but small and unrepeated.
-
The cost-efficiency conclusion rests on narrow data. Cost is measured well, but the three-harness cost comparison covers two vendor-API models on the 45 hard tasks, one run each, so whether the cheaper harness is equally effective there is unresolved. The first-call mechanism is reconstructed on GLM, while on DeepSeek the bill is driven by step count instead, so "set at the first call" is overstated.
-
Novelty is incremental: Vats & Golev (2026) already report small pass-rate gaps with large token differences across production harnesses, and Bjarnason et al. (2026) measure SWE-bench rerun noise.
-
Only one benchmark is studied in depth; the Terminal-Bench check has one model and no reruns.
Questions for authors
- What justifies the ±5-point equivalence margin, and what are the verdicts at ±2 and ±3?
- How do the OpenCode results change with equal wall-clock limits, and with a larger per-response output cap for both harnesses?
- Can the hard set be rerun several times per arm for all models, including GLM and HY4?
- How do results look for a frontier model run in all three harnesses?
- What are cost per solved task and pass rate under equal token or dollar budgets?
3 comments
Response to Reviewer 49cb (2/2)
Concern (4): cost
You were concerned that the cost conclusion rested on two vendor models with one run each. It now rests on five models and two task sets, with reruns, and the wider data change the conclusion: the cost gap between harnesses depends on the model, from under 10% (DeepSeek-V4.1-Flash pool447, Claude Opus 5.5 hard45) to 3 to 4 times (GLM-5.3-Flash). This range replaces "up to 3×" in the revision.
The table gives mean cost per trial, computed as in the submission (tokens times the submission's price list, with DeepSeek-V4.1-Flash at DeepSeek-V4-Flash prices). Claude Opus 5.5 uses Anthropic's list prices, which are outside that list. Costs compare within a row only. The last column is the ratio for cost per solved task, pooled over runs.
| Model, task set | Currency | Claude Code | mini-SWE-agent | OpenCode | Max ÷ min | Per solved task |
|---|---|---|---|---|---|---|
| GLM-5.3-Flash, hard45, run 1 / run 2 | CNY | 1.81 / 2.09 | 0.97 / 1.06 | 0.56 / 0.52 | 3.2× / 4.0× | 3.15× |
| HY4-Preview, hard45, run 1 / run 2 | CNY | 2.86 / 2.42 | 4.13 / 3.20 | 2.41 / 1.63 | 1.7× / 2.0× | 1.6× |
| DeepSeek-V4-Flash, hard45, run 1 | CNY | 0.98 | 1.02 | 0.66 | 1.5× | 1.3× |
| DeepSeek-V4.1-Flash, hard45, 2 runs | CNY | 0.31 | 0.35 | 0.41 | 1.3× | 1.3× |
| DeepSeek-V4.1-Flash, pool447, 2 runs | CNY | 0.143 | 0.148 | 0.147 | 1.03× | 1.05× |
| Claude Opus 5.5, hard45 | USD | 0.42 | 0.39 | 0.39 | 1.08× | 1.14× |
The first-run gaps were not accidents of one run: the second runs on GLM-5.3-Flash and HY4-Preview reproduce them in the same order. Where it is largest, on GLM-5.3-Flash, Claude Code passes 41/90 against OpenCode's 36/90 over two runs, so the extra spending there did not detectably buy passes.
"Set at the first call" was overstated, as you say. In Table A4 the fixed cost of each call is at most about 23% of per-step input (the highest, Claude Code's). The revision replaces that sentence with three terms: the preamble sent with every step, the content kept in history, and the number of steps.
Question 5: cost per solved task and equal budgets
Both analyses you asked for give the same reading: the order of the harnesses follows what they spend.
Cost per solved task follows cost per trial because pass rates are close. The largest gap is on GLM-5.3-Flash: over two runs Claude Code costs 4.28 CNY per solved task and OpenCode 1.36, and their 95% intervals do not overlap. On the 445-task pool the three harnesses are within 5%.
Equal budgets. We rescored existing trials, counting any trial above a common budget as a failure. On DeepSeek-V4.1-Flash pool447 (run 1) the three harnesses are within 0.2 points without a budget. With the budget at the pooled median of output tokens, Claude Code passes 53.5%, OpenCode 49.2% and mini-SWE-agent 33.7%. Once a budget binds, the order follows how many output tokens each harness uses, which says which harness spends less, not which solves more. The revision puts cost per solved task in the main text and the equal-budget curves in the appendix, labelled as post-hoc truncation.
Concern (5): novelty
We see the question as still open, and this review round reflects it: of the same conclusion, one reviewer finds it surprising and contrary to many results, and you find it reported before. The two earlier papers do not settle it.
- Vats & Golev (2026) have one run per cell and no rerun baseline, and the 95% intervals of their differences are 18 to 20 points wide. Most cover zero, and at that width an interval covering zero cannot separate no difference from too little resolution.
- Bjarnason et al. (2026) measure SWE-bench rerun noise and do not swap harnesses.
The experiments you suggested then showed, on SWE-bench Verified and against a rerun baseline, that the effect of a harness on pass rate depends on the model it is paired with. OpenCode trails by 8 to 9 points on Qwen3.8-27B and leads by 14 on DeepSeek-V4.1-Flash hard45, so the question is less simple than we first assumed.
Our paper adds: (i) Rerun noise as the yardstick for a harness swap, on the same tasks. (ii) A pool large enough to give a bound (445 tasks, four models). (iii) Cost under one price list, with its decomposition. (iv) The leakage audit of web-enabled runs.
Concern (6): one benchmark
If the result were specific to repository repair, a different kind of task should separate the harnesses. We added two benchmarks of other kinds, with DeepSeek-V4.1-Flash in all three harnesses and two runs each, and we detect no difference on either. On SpreadsheetBench Verified the 90% intervals exclude gaps beyond 2 points, and on DABstep the harnesses changed order between the two runs.
We believe the new experiments substantially strengthen the paper's evidence on both pass rate and cost, and we would be grateful if you would reconsider your rating.
Follow-up on Our Response
Dear Reviewer 49cb,
The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.
In brief: the revised claim is the sentence you found supported, and the large task pool now has four models in all three harnesses. The checks you asked for are run: OpenCode with equal wall clock and with a larger output cap, and Claude Opus 5.5 through the native interface in all three harnesses. Cost per solved task is reported, and the cost claim is now a range that depends on the model.
If anything remains unclear or unresolved, please let us know and we will respond before the deadline.
Thank you again for your time.
Response to Reviewer 49cb (1/2)
We thank Reviewer 49cb. Questions 1 to 4 each became an experiment. All results are in the table in our general response (2/2).
Concern (1), Questions 1 and 3: power and the claim
You wrote "'Not shown to differ' is supported; 'barely moves' is not". We agree, and the revised claim is the sentence you find supported. Beyond it, where the task pool is large enough, the new data give bounds.
More models and pairs. pool447 now has four models in all three harnesses. On DeepSeek-V4.1-Flash, run twice, the 90% intervals of all three pairs exclude gaps beyond 3.2 points. Averaged over the four models, Claude Code and mini-SWE-agent differ by 0.2 points (90% interval [−0.9, +1.2]); per model the difference runs from −1.8 to about +2.
Models without reruns. GLM-5.3-Flash, HY4-Preview and DeepSeek-V4.1-Flash now have two runs in all three harnesses on hard45. The widest spread in the submission (HY4-Preview, 5 tasks between harnesses) did not recur: all three pass 24 in the second run.
A consistent effect. You are right that matching flip rates cannot exclude one. A consistent effect would appear as an average gap across models. Pooling seven models on hard45, the 95% intervals of all three pairs cover zero and exclude an average gap above 5 points. Single-model differences instead run from −14.4 to +8.9 points, and for the two pairs with OpenCode this disagreement exceeds sampling error (heterogeneity p = 0.01 and 0.02). So no harness leads on average, and OpenCode's position changes with the model.
Margins. δ = 5 was not preset: we tightened an initial ±10 after pool447 was run, to that pool's resolution (about 5.2 points at 80% power). In the table, "yes" means the whole 90% interval lies within ±δ, so a gap larger than δ is excluded.
Most single-model comparisons fail at ±2 and ±3 because 445 tasks are underpowered for a 2-point gap on one model (minimum detectable gap 3.7 points on DeepSeek-V4.1-Flash). The tighter bound holds for the average over models, not for each model. The revision drops the preset-margin wording and reports this table.
Concern (2), Question 2: the OpenCode effect
We agree: limit-ended trials are part of the harness outcome, and OpenCode's deficit on Qwen3.8-27B is a harness effect on pass rate. The revision puts it in the claim itself, and your two checks narrow its cause.
Equal wall clock. We reran OpenCode on hard45 with Claude Code's 36,000 s, twice (90 trials). Trials ended by the wall clock fall from 12 to none, yet passes are 32 of 89 scored trials against 31 of 89 before (one trial per side could not be scored). Had the shorter wall clock caused the deficit, passes should have risen, so it does not explain it.
Larger output cap. On the 73 pool447 tasks where any OpenCode trial in the submission had ended on its cap, we alternated a 32k and a 131k cap, two runs each. Passes are 86/146 and 99/146 (p = 0.076), consistent with truncation losing tasks, short of significance. We ran OpenCode only, since in the submission 34 of its trials on this pool ended on the cap (33 failed), against 1 for Claude Code and none for mini-SWE-agent.
The same harness leads elsewhere. On DeepSeek-V4.1-Flash, where the three harnesses share timeout and cap settings, OpenCode leads mini-SWE-agent by 14 points on hard45 over two runs and by 1.8 points on pool447. So the harness effect you point to exists and depends on the model.
Concern (3), Question 4: conditions and a frontier model
We ran Claude Opus 5.5 in all three harnesses through the native Anthropic interface. This is the setting most favourable to Claude Code: if the compatibility layer had held it back, it should lead here. On hard45 it does not lead (36, 37, 36 in three runs, against 37 and 38). On pool447 (the 391 tasks all three completed) it is about 2 points higher (95.1% against 93.1% and 93.4%; p = 0.06 and 0.09, or 0.17 and 0.18 after Holm correction), and the 90% intervals put the lead between 0 and 4 points. So with the native interface Claude Code's lead is at most 4 points on pool447 and absent on hard45.
On Grep: whenever the shell is available, Claude Code's tool list has no separate Grep tool for any model, Anthropic or not, so search runs through the shell. The revision corrects the appendix sentence and its configuration table.
The revision adds a table of the unequal conditions with their checks. Cost, novelty and the single benchmark follow in (2/2).