acceptodds
Under review as a conference paper at ICLR 2027

What Does a Harness Buy? Tokens, Mostly.

Abstract

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Much of the day-to-day engineering of these agents happens in the harness. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. This paper measures it. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. The answer to the title question is: not pass rate. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3× across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider’s price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

Primary area:
datasets and benchmarks
Keywords:
coding agents, agent harness, SWE-bench Verified, evaluation methodology, measurement noise, data contamination, paired statistics, inference cost

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_49cbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how much the harness (system prompt, tools, context management) changes a coding agent's results when the model is fixed. It runs five models through Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified, with reruns to measure run-to-run noise. Its conclusions are that harness swaps move pass rate about as much as reruns do, that the one clear effect is OpenCode losing tasks, and that harnesses mainly change cost, by up to 3×, driven by first-call overhead, context growth, and step count.

Critical strengths
  • An important, practical question, and the right design for it: same model, same tasks, reruns as a yardstick, and paired tests.
  • Careful, honest engineering: pinned versions, offline runs (after an audit found 71% of early web-enabled DeepSeek and Qwen trials fetched the upstream fix), and failure modes traced in the trajectories.
  • The cost decomposition (preamble × steps, with growth per step) is clear and useful, and the power analysis shows readers what a 45-task comparison can and cannot resolve.
Critical weaknesses

Main concerns

  1. The headline "not pass rate" is mostly a lack of power, not evidence of no effect. Equivalence is shown only for Claude Code vs. mini-SWE-agent, on the two locally served Qwen models, at a ±5-point margin wider than the gains vendors advertise. The other three models are tested only on the 45 hard tasks, where no gap is well powered, yet some point estimates there are as large as the OpenCode gap the paper does call real. The noise floor is also thin: two of the models have no reruns, and the task-level replication test rests on very few tasks. Matching flip rates (13% for reruns and harness swaps) show similar churn, not the absence of a consistent effect. "Not shown to differ" is supported; "barely moves" is not.

  2. The paper's own significant results are harness effects on pass rate. OpenCode underperforms by about 8 to 9 points on Qwen3.8-27B, and both Qwen3.8 gaps survive Holm correction, which undercuts the broad framing even if the paper presents OpenCode as an exception. Removing limit-ended trials roughly halves the gap (Table A7), but limit-ended trials are themselves part of the harness outcome, and the recovery experiment did not establish the proposed mechanism (Appendix H).

  3. Some comparisons are unequal, and the scope is narrower than the title. On the locally served Qwen arms OpenCode gets 5,760 s of wall clock versus 36,000 s for Claude Code; dropping the affected tasks does not shrink its gap (Section 4.3), but it weakens the design. More importantly for scope, for every non-Anthropic model, Claude Code runs through a compatibility layer (without its Grep tool), and Opus 5 is never run in another harness, so no Anthropic frontier model is compared across harnesses. The DeepSeek home-harness check is relevant but small and unrepeated.

  4. The cost-efficiency conclusion rests on narrow data. Cost is measured well, but the three-harness cost comparison covers two vendor-API models on the 45 hard tasks, one run each, so whether the cheaper harness is equally effective there is unresolved. The first-call mechanism is reconstructed on GLM, while on DeepSeek the bill is driven by step count instead, so "set at the first call" is overstated.

  5. Novelty is incremental: Vats & Golev (2026) already report small pass-rate gaps with large token differences across production harnesses, and Bjarnason et al. (2026) measure SWE-bench rerun noise.

  6. Only one benchmark is studied in depth; the Terminal-Bench check has one model and no reruns.

Questions for authors
  1. What justifies the ±5-point equivalence margin, and what are the verdicts at ±2 and ±3?
  2. How do the OpenCode results change with equal wall-clock limits, and with a larger per-response output cap for both harnesses?
  3. Can the hard set be rerun several times per arm for all models, including GLM and HY4?
  4. How do results look for a frontier model run in all three harnesses?
  5. What are cost per solved task and pass rate under equal token or dollar budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 49cb (1/2)

We thank Reviewer 49cb. Questions 1 to 4 each became an experiment. All results are in the table in our general response (2/2).

Concern (1), Questions 1 and 3: power and the claim

You wrote "'Not shown to differ' is supported; 'barely moves' is not". We agree, and the revised claim is the sentence you find supported. Beyond it, where the task pool is large enough, the new data give bounds.

More models and pairs. pool447 now has four models in all three harnesses. On DeepSeek-V4.1-Flash, run twice, the 90% intervals of all three pairs exclude gaps beyond 3.2 points. Averaged over the four models, Claude Code and mini-SWE-agent differ by 0.2 points (90% interval [−0.9, +1.2]); per model the difference runs from −1.8 to about +2.

Models without reruns. GLM-5.3-Flash, HY4-Preview and DeepSeek-V4.1-Flash now have two runs in all three harnesses on hard45. The widest spread in the submission (HY4-Preview, 5 tasks between harnesses) did not recur: all three pass 24 in the second run.

A consistent effect. You are right that matching flip rates cannot exclude one. A consistent effect would appear as an average gap across models. Pooling seven models on hard45, the 95% intervals of all three pairs cover zero and exclude an average gap above 5 points. Single-model differences instead run from −14.4 to +8.9 points, and for the two pairs with OpenCode this disagreement exceeds sampling error (heterogeneity p = 0.01 and 0.02). So no harness leads on average, and OpenCode's position changes with the model.

Margins. δ = 5 was not preset: we tightened an initial ±10 after pool447 was run, to that pool's resolution (about 5.2 points at 80% power). In the table, "yes" means the whole 90% interval lies within ±δ, so a gap larger than δ is excluded.

Comparison on pool44790% interval±2±3±5
Qwen3.6, Claude Code − mini-SWE-agent[−4.7, +1.1]nonoyes
Qwen3.8, Claude Code − mini-SWE-agent[−3.7, +1.0]nonoyes
DeepSeek-V4.1-Flash, Claude Code − OpenCode[−1.8, +1.4]yesyesyes
DeepSeek-V4.1-Flash, other two pairs[−0.0, +3.2], [−3.2, −0.4]nonoyes
Four-model average, Claude Code − mini-SWE-agent[−0.9, +1.2]yesyesyes

Most single-model comparisons fail at ±2 and ±3 because 445 tasks are underpowered for a 2-point gap on one model (minimum detectable gap 3.7 points on DeepSeek-V4.1-Flash). The tighter bound holds for the average over models, not for each model. The revision drops the preset-margin wording and reports this table.

Concern (2), Question 2: the OpenCode effect

We agree: limit-ended trials are part of the harness outcome, and OpenCode's deficit on Qwen3.8-27B is a harness effect on pass rate. The revision puts it in the claim itself, and your two checks narrow its cause.

Equal wall clock. We reran OpenCode on hard45 with Claude Code's 36,000 s, twice (90 trials). Trials ended by the wall clock fall from 12 to none, yet passes are 32 of 89 scored trials against 31 of 89 before (one trial per side could not be scored). Had the shorter wall clock caused the deficit, passes should have risen, so it does not explain it.

Larger output cap. On the 73 pool447 tasks where any OpenCode trial in the submission had ended on its cap, we alternated a 32k and a 131k cap, two runs each. Passes are 86/146 and 99/146 (p = 0.076), consistent with truncation losing tasks, short of significance. We ran OpenCode only, since in the submission 34 of its trials on this pool ended on the cap (33 failed), against 1 for Claude Code and none for mini-SWE-agent.

The same harness leads elsewhere. On DeepSeek-V4.1-Flash, where the three harnesses share timeout and cap settings, OpenCode leads mini-SWE-agent by 14 points on hard45 over two runs and by 1.8 points on pool447. So the harness effect you point to exists and depends on the model.

Concern (3), Question 4: conditions and a frontier model

We ran Claude Opus 5.5 in all three harnesses through the native Anthropic interface. This is the setting most favourable to Claude Code: if the compatibility layer had held it back, it should lead here. On hard45 it does not lead (36, 37, 36 in three runs, against 37 and 38). On pool447 (the 391 tasks all three completed) it is about 2 points higher (95.1% against 93.1% and 93.4%; p = 0.06 and 0.09, or 0.17 and 0.18 after Holm correction), and the 90% intervals put the lead between 0 and 4 points. So with the native interface Claude Code's lead is at most 4 points on pool447 and absent on hard45.

On Grep: whenever the shell is available, Claude Code's tool list has no separate Grep tool for any model, Anthropic or not, so search runs through the shell. The revision corrects the appendix sentence and its configuration table.

The revision adds a table of the unequal conditions with their checks. Cost, novelty and the single benchmark follow in (2/2).

AuthorsOpenReview ↗

Response to Reviewer 49cb (2/2)

Concern (4): cost

You were concerned that the cost conclusion rested on two vendor models with one run each. It now rests on five models and two task sets, with reruns, and the wider data change the conclusion: the cost gap between harnesses depends on the model, from under 10% (DeepSeek-V4.1-Flash pool447, Claude Opus 5.5 hard45) to 3 to 4 times (GLM-5.3-Flash). This range replaces "up to 3×" in the revision.

The table gives mean cost per trial, computed as in the submission (tokens times the submission's price list, with DeepSeek-V4.1-Flash at DeepSeek-V4-Flash prices). Claude Opus 5.5 uses Anthropic's list prices, which are outside that list. Costs compare within a row only. The last column is the ratio for cost per solved task, pooled over runs.

Model, task setCurrencyClaude Codemini-SWE-agentOpenCodeMax ÷ minPer solved task
GLM-5.3-Flash, hard45, run 1 / run 2CNY1.81 / 2.090.97 / 1.060.56 / 0.523.2× / 4.0×3.15×
HY4-Preview, hard45, run 1 / run 2CNY2.86 / 2.424.13 / 3.202.41 / 1.631.7× / 2.0×1.6×
DeepSeek-V4-Flash, hard45, run 1CNY0.981.020.661.5×1.3×
DeepSeek-V4.1-Flash, hard45, 2 runsCNY0.310.350.411.3×1.3×
DeepSeek-V4.1-Flash, pool447, 2 runsCNY0.1430.1480.1471.03×1.05×
Claude Opus 5.5, hard45USD0.420.390.391.08×1.14×

The first-run gaps were not accidents of one run: the second runs on GLM-5.3-Flash and HY4-Preview reproduce them in the same order. Where it is largest, on GLM-5.3-Flash, Claude Code passes 41/90 against OpenCode's 36/90 over two runs, so the extra spending there did not detectably buy passes.

"Set at the first call" was overstated, as you say. In Table A4 the fixed cost of each call is at most about 23% of per-step input (the highest, Claude Code's). The revision replaces that sentence with three terms: the preamble sent with every step, the content kept in history, and the number of steps.

Question 5: cost per solved task and equal budgets

Both analyses you asked for give the same reading: the order of the harnesses follows what they spend.

Cost per solved task follows cost per trial because pass rates are close. The largest gap is on GLM-5.3-Flash: over two runs Claude Code costs 4.28 CNY per solved task and OpenCode 1.36, and their 95% intervals do not overlap. On the 445-task pool the three harnesses are within 5%.

Equal budgets. We rescored existing trials, counting any trial above a common budget as a failure. On DeepSeek-V4.1-Flash pool447 (run 1) the three harnesses are within 0.2 points without a budget. With the budget at the pooled median of output tokens, Claude Code passes 53.5%, OpenCode 49.2% and mini-SWE-agent 33.7%. Once a budget binds, the order follows how many output tokens each harness uses, which says which harness spends less, not which solves more. The revision puts cost per solved task in the main text and the equal-budget curves in the appendix, labelled as post-hoc truncation.

Concern (5): novelty

We see the question as still open, and this review round reflects it: of the same conclusion, one reviewer finds it surprising and contrary to many results, and you find it reported before. The two earlier papers do not settle it.

  • Vats & Golev (2026) have one run per cell and no rerun baseline, and the 95% intervals of their differences are 18 to 20 points wide. Most cover zero, and at that width an interval covering zero cannot separate no difference from too little resolution.
  • Bjarnason et al. (2026) measure SWE-bench rerun noise and do not swap harnesses.

The experiments you suggested then showed, on SWE-bench Verified and against a rerun baseline, that the effect of a harness on pass rate depends on the model it is paired with. OpenCode trails by 8 to 9 points on Qwen3.8-27B and leads by 14 on DeepSeek-V4.1-Flash hard45, so the question is less simple than we first assumed.

Our paper adds: (i) Rerun noise as the yardstick for a harness swap, on the same tasks. (ii) A pool large enough to give a bound (445 tasks, four models). (iii) Cost under one price list, with its decomposition. (iv) The leakage audit of web-enabled runs.

Concern (6): one benchmark

If the result were specific to repository repair, a different kind of task should separate the harnesses. We added two benchmarks of other kinds, with DeepSeek-V4.1-Flash in all three harnesses and two runs each, and we detect no difference on either. On SpreadsheetBench Verified the 90% intervals exclude gaps beyond 2 points, and on DABstep the harnesses changed order between the two runs.

We believe the new experiments substantially strengthen the paper's evidence on both pass rate and cost, and we would be grateful if you would reconsider your rating.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 49cb,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the revised claim is the sentence you found supported, and the large task pool now has four models in all three harnesses. The checks you asked for are run: OpenCode with equal wall clock and with a larger output cap, and Claude Opus 5.5 through the native interface in all three harnesses. Cost per solved task is reported, and the cost claim is now a range that depends on the model.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

Reviewer_bxRAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors test five models with three coding-agent harnesses, Claude Code, mini-SWE-agent and OpenCode, on SWE-bench Verified. They keep the model fixed when comparing harnesses and repeat runs to check how much the results vary on their own. For the two models tested on the 447-task pool, Claude Code and mini-SWE-agent are equivalent within five percentage points. On the 45 hard tasks, changing the harness and rerunning the same harness both give a median task-level flip rate of 13%.

Critical strengths

I think the question is important for agent research, and the experiments are interesting.

Critical weaknesses

I don't agree with the claim that harness effects are “rarely measured.” Agents' Last Exam already compares different harnesses with the same model. So the motivation, at least the way it is written now, is not very convincing.

Also, I think the experiments are too limited. The settings are not fully consistent across harnesses. This makes it hard to trust the comparison of the production harnesses. The writing and figures need improvement as well.

Questions for authors

The writing needs more work. Some parts are hard to follow, and the figures are not clear enough.

The claim that “what is rarely measured is how much the harness itself moves the score when the model is held fixed” seems too strong. There are already papers doing this, including Agents' Last Exam. I think the authors need to explain the difference from these papers more clearly. The current motivation does not convince me.

The experiments are not extensive enough. Also, some settings differ across harnesses, so the comparison is not fully controlled. This is a concern for the production-harness results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bxRA

We thank Reviewer bxRA. Your comment on breadth and consistency led to two new benchmarks and a new table of checks, and your pointer to Agents' Last Exam filled a gap in our related work.

1. Related work: "rarely measured" and Agents' Last Exam

Agents' Last Exam (Sun et al., 2026) fixes a model and varies the harness in its Table 1. Our abstract's sentence did not mention existing work of this kind and should be more precise, and we had missed this citation. The revision rewrites the sentence and discusses that paper in Related Work, as evidence in the same direction. With GPT-5.5 fixed, its five harnesses span 4.9 points, and with one harness fixed its twelve models span 16.8 points.

The question our paper asks is still open after that work, whose design goal is to rank systems. Its harness-swap groups are mostly single runs (the group that fixes Claude Opus 4.7 across three harnesses, with a 7.2-point spread, has no repeated cell), and the harness differences come without intervals, tests, or a rerun reference. So that design leaves open how much of a 4.9 or 7.2-point spread is the harness and how much is rerun noise.

Our paper measures that part. We rerun each configuration unchanged and compare, on the same tasks, how many tasks flip (pass in one run, fail in the other) under a harness swap and under a rerun. The premise therefore holds in a more precise form: existing measurements are almost all single runs, and whether a harness swap moves the score more than a rerun does had not been measured.

2. Breadth of evaluation and consistency across harnesses

Breadth. You asked for a broader evaluation. The submission's main results rest on 4,237 scored trials, and we added about 8,400, roughly twice as many: two new models in all three harnesses on SWE-bench Verified (DeepSeek-V4.1-Flash, run twice, and the frontier model Claude Opus 5.5), second runs for the two models that had one, two new benchmarks, and two checks on OpenCode's deficit. The full table is in our general response.

The two benchmarks test whether the result is specific to repository repair. If it were, a different kind of task should separate the harnesses. SpreadsheetBench Verified (400 spreadsheet tasks) and DABstep (150 data-analysis questions) each ran in all three harnesses, twice, with DeepSeek-V4.1-Flash. We detect no difference on either: on the first the 90% intervals exclude gaps beyond 2 points, and on the second the three harnesses changed order between the two runs.

Consistency. Some settings did differ across harnesses in the submission, and our text did not set them out in one place. The revision adds the table below as Table A9 in the appendix, with a check of each condition.

Condition that differedCheckResult
On Qwen3.8-27B hard45 OpenCode had 5,760 s of wall clock, Claude Code 36,000 sOpenCode rerun at 36,000 s (Qwen3.8-27B, hard45, 2 runs)Timeouts 12 → 0, passes 31/89 → 32/89: wall clock does not explain the deficit
OpenCode does not continue a response cut by its output capCap 32k vs. 131k on the 73 affected tasks, 2 runs eachPasses 86/146 → 99/146 (p = 0.076): consistent with truncation losing tasks, short of significance
For non-Anthropic models Claude Code runs through a compatibility layerClaude Opus 5.5 run natively in all three harnesseshard45 (45 hard tasks): Claude Code 36, 37, 36; mini-SWE-agent 37; OpenCode 38. pool447 (the other tasks, 391 done by all three): 95.1%, 93.1%, 93.4%
Timeout and output cap (rows 1 and 2)DeepSeek-V4.1-Flash, same soft timeout in all three harnesses and same 32k output cap in Claude Code and OpenCode (pool447, 2 runs)83.4%, 81.8%, 83.6%: within 2 points under equal settings

Percentages are ordered Claude Code, mini-SWE-agent, OpenCode. Row 3 shows that with the compatibility layer removed any lead for Claude Code is small: about 2 points on pool447, not significant after Holm correction and at most 4 points, and none on hard45.

3. Writing and figures

Readability was your first question, and we take it seriously: a reader who cannot follow the figures cannot check the claims. The uploaded revision makes the changes below, with changed text in blue.

  • The first sentence of every section states only that section's conclusion.
  • Figure 1b becomes one pooled figure in the main text and a full table in the appendix (Table A8). Figure 1a loses its connecting lines and its legend moves outside the plot.
  • The appendix gains the table above (Table A9), and Table 1 now lists every run of every configuration.
  • The abstract and introduction sentences that our replies commit to changing are rewritten, "rarely measured" among them.

We believe the new experiments and the discussion of Agents' Last Exam substantially strengthen the paper, and we would be grateful if you would reconsider your rating.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer bxRA,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the revision discusses Agents' Last Exam in Related Work, as evidence in the same direction. The evaluation is broader: on two benchmarks of other kinds, spreadsheet editing and data analysis, we detect no difference between the three harnesses. Each setting that differed across harnesses now has its own check (Table A9), and Figure 1 is redrawn.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

Thank you for the response and additional experiments. I think these address most of my concerns, and I will raise my rating to 3. I would still appreciate a clearer explanation of how far the conclusions can be generalized beyond the models and settings tested.

AuthorsOpenReview ↗

Reply to Reviewer bxRA

Thank you for reading the response and for raising your rating.

What we expect to hold beyond our runs is the pattern, and the reason lies in how the agents behave. In every harness it is the same model that reads the code and decides the fix. After a harness swap the model edits about the same files as on a rerun of the same configuration, and it takes a different model to change them. Outcomes agree as well. With Claude Opus 5.5, 373 of 391 tasks end the same way in all three harnesses, and of the 19 to 27 tasks each harness fails, 15 are the same tasks. A harness therefore changes the outcome only in particular pairings, for example when it does not continue a response that was cut off. These pairings differ from model to model, so which harness scores higher on a new model has to be measured on that model.

What we do not extend is the size of the gaps. The bounds of 2 to 5 points and the cost ratios are measurements on the models and tasks we ran.

The final version will say this in the Discussion.

Thank you again. Your question led us to state more clearly which part of the conclusion carries over.

Reviewer_16QSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how much of a coding agent's score can be attributed to the harness, not the model. The paper tests five models with Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified. The main conclusion is the harness has little effect on accuracy, but a larger effect on cost.

Critical strengths

The authors breakdown the headline cost result very carefully: Table A4 separates the fixed per-call preamble (Claude Code / mini-SWE-agent / OpenCode: 16,581 / 829 / 7,025 tokens), per-step growth (937 / 648 / 681) and step count (88 / 80 / 55) on GLM-5.3-Flash; per-call tool returns are near-identical (means 1,237 / 1,186 / 1,103 characters), even taking into account different token costs from different model developers.

Critical weaknesses

My main gripe with the paper is the lack of power due to lack of replications: at n = 45, the 50%-power minimum detectable gap is 0.13. The authors are also claiming a (at least to me) surprising result: the harness doesn't matter for accuracy, when there're lots of results that show otherwise, so the burden of proof on the authors is high. I will raise my score if the authors address this concern and meet my burdne of proof (eg, by running more replicates).

Questions for authors
  1. [clarification] §3 fixes δ = 5 points without saying why (p. 5), and the Qwen3.6 Claude Code−mini 90% interval, [−4.7, +1.1] (Appendix F, p. 17), would fall outside a 4-point margin. Was δ fixed in advance, and what is the smallest harness effect you intend to exclude?

  2. [clarification] §4.1 says "Claude Code is at most about one point ahead" (p. 5), while Table 1's point estimates put it 1.8 and 1.4 points behind. Are you reporting the 90% upper bound (+1.1, +1.0)?

  3. [clarification] On hard45, two of the five ladder models have no second run in any arm. Would adding one change any spread or flip-rate claim, and if not, should "across five models" be narrowed to the Qwen cells?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 16QS

We thank Reviewer 16QS. Your request for more replicates led to two more models on the large task pool and to second runs for the two models that had none.

Replicates and power

Your concern was that a surprising conclusion rested on too few replicates and too little power. On the large task pool (pool447, the 447 usable SWE-bench Verified tasks outside the 45-task hard subset, hard45), the submission had two models and could limit the gap for one pair of harnesses only: Claude Code and mini-SWE-agent were shown to be at most 5 points apart, and the two pairs with OpenCode had no such limit. The pool now has four models in all three harnesses. On DeepSeek-V4.1-Flash, run twice, every pair has a limit: none is more than 3.2 points apart. Averaged over the four models, Claude Code and mini-SWE-agent differ by 0.2 points (95% interval [−1.1, +1.4]); per model the difference runs from −1.8 to about +2.

Paired differences on DeepSeek-V4.1-Flash (445 scorable tasks, both runs pooled):

PairDifference (points)90% interval
Claude Code − mini-SWE-agent+1.6[−0.0, +3.2]
Claude Code − OpenCode−0.2[−1.8, +1.4]
mini-SWE-agent − OpenCode−1.8[−3.2, −0.4]

On this model a harness swap flips 6.9% to 7.8% of tasks (a flip is a task that passes in one run and fails in the other), and rerunning the same configuration flips 6.3% to 7.6%, so a harness swap changes about as many outcomes as a rerun. A single run can also mislead: mini-SWE-agent and OpenCode were 0.2 points apart in run 1 and 3.4 in run 2 (Holm-corrected p = 0.024); pooled, the gap is 1.8 points and not significant after Holm correction.

"Lots of results that show otherwise"

We agree the burden of proof is high and answer in three steps. First, the example above shows that the size of a gap read from one run is not stable. Second, the large published spreads our introduction cites (23.8 and 27.4 points) partly measure how each harness was wired to the benchmark: in the latter one harness scores 19.1% with a minimal adapter and 73.4% with the full one. Agents' Last Exam (Sun et al., 2026) finds far less: 4.9 points over five harnesses with the model fixed, against 16.8 points over twelve models with the harness fixed.

Third, behaviour agrees. On the five models with reruns in all three harnesses, we compared which source files two trials end up editing. By Jaccard similarity, reruns of one configuration score 0.796 and a harness swap 0.786, with no difference detected; a model swap scores 0.735, detectably less. So after a harness swap the agent edits about the same files as on a rerun, and it takes a different model to change where it ends up.

Our claim is also narrower than our wording made it appear, and the new data sharpen it. No harness leads consistently across models, yet harness effects on pass rate exist in both directions. OpenCode trails by 8 to 9 points on Qwen3.8-27B (pool447). On DeepSeek-V4.1-Flash hard45 it is the highest, with 27 and 27 against 21 and 20 for mini-SWE-agent, a 14-point lead. So the effect of a harness on pass rate depends on the model it is paired with, and "which harness is better" has no answer apart from the model. The revision states this.

Q1. Was δ = 5 fixed in advance?

No. The initial design document preset ±10 points, and we tightened it to 5 after pool447 was run, to match that pool's resolution (about 5.2 points at 80% power). So 5 is the finest gap one paired comparison on this pool can resolve, not a statement about the smallest effect that matters.

Your reading is correct: the Qwen3.6 interval [−4.7, +1.1] and the Qwen3.8 interval [−3.7, +1.0] hold at ±5 and fail at ±3 and ±2, so these data cannot exclude Claude Code trailing by 3 to 4 points. One comparison on 445 tasks is underpowered for a 2-point gap (minimum detectable gap 3.7 points on DeepSeek-V4.1-Flash). The four-model average above holds at ±2, as a bound on the average gap and not on each model. The revision no longer presents ±5 as a preset margin and reports verdicts at ±2, ±3 and ±5 per model.

Q2. "At most about one point ahead"

Yes, and the sentence was unclear. It becomes: "point estimates put Claude Code 1.8 and 1.4 points behind mini-SWE-agent, and the upper ends of the 90% intervals, +1.1 and +1.0, bound how far it could be ahead."

Q3. The two models without a second run

GLM-5.3-Flash and HY4-Preview now have a second run in all three harnesses, and the conclusions do not change. HY4-Preview had the widest spread in the submission, 5 tasks between harnesses. In its second run all three pass 24, so that spread did not recur. With these runs added, the median rerun and harness-swap flip rates on hard45 are both still 13.3%.

We believe the added replicates substantially strengthen the evidence and meet the burden of proof you describe, and we would be grateful if you would reconsider your rating.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 16QS,

The discussion period closes on Nov 18 (AoE), so we are writing to check whether our response addressed your concerns.

In brief: the replicates you asked for are now in. The large task pool has four models in all three harnesses instead of two, and on DeepSeek-V4.1-Flash, run twice in every harness, no pair of harnesses is more than 3.2 points apart. GLM-5.3-Flash and HY4-Preview now have a second run in every harness, and the widest spread between harnesses in the submission did not recur.

If anything remains unclear or unresolved, please let us know and we will respond before the deadline.

Thank you again for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank the three reviewers. The same conclusion was read in two ways: Reviewer 16QS finds it surprising, since many results suggest the harness changes accuracy, and Reviewer 49cb finds that earlier work already reports something similar. We take this as a sign that the question is not settled, and that settling it takes resolution: more tasks, more models, and reruns.

What we added. The submission's main results rest on 4,237 scored trials. We added about 8,400 more, roughly twice as many, in four groups:

  • Two new models in all three harnesses on both task sets: DeepSeek-V4.1-Flash, run twice, and the frontier model Claude Opus 5.5.
  • Second runs in all three harnesses for GLM-5.3-Flash and HY4-Preview, which had one run each.
  • Two new benchmarks that are not repository repair, each run twice.
  • Two checks on why OpenCode trails on one model: equal wall clock and a larger output cap.

All results are in the table in our second comment (2/2), with additions in bold.

Terms: hard45 is the 45-task hard subset of SWE-bench Verified and pool447 is the other 447 usable tasks. A rerun repeats a configuration unchanged. Reruns are the yardstick: a harness swap has to move the score more than a rerun does.

1. A single run can show a gap that the next run does not. On DeepSeek-V4.1-Flash pool447 the three harnesses were within 0.2 points in run 1. In run 2, mini-SWE-agent trailed OpenCode by 3.4 points (Holm-corrected p = 0.024 within that run). This bears on Reviewer 16QS's remark that many results show otherwise: the size of a gap read from one run is not stable. Pooled over both runs, the gap narrows to 1.8 points, no pair is significant after Holm correction, and the 90% intervals of all three pairs exclude gaps beyond 3.2 points.

2. On a frontier model run natively, any lead for Claude Code is small. Reviewer 49cb noted that no frontier model was compared across harnesses and that Claude Code ran through a compatibility layer on non-Anthropic models. Claude Opus 5.5 runs natively in all three harnesses, so if the compatibility layer had held Claude Code back, it should lead clearly here. On hard45 it does not lead: Claude Code passes 36, 37 and 36 in three runs, mini-SWE-agent 37 and OpenCode 38. On pool447 (the 391 tasks all three completed) the pass rates are 95.1%, 93.1% and 93.4%. Claude Code is about 2 points ahead there, not significant after Holm correction, and the 90% intervals put its lead between 0 and 4 points. So in the setting most favourable to Claude Code, its lead is at most 4 points on pool447 and absent on hard45.

3. Two benchmarks of other kinds show the same picture. Reviewers bxRA and 49cb noted that one benchmark was studied in depth. If the result depended on that benchmark, a different kind of task should separate the harnesses. On SpreadsheetBench Verified and DABstep, both run with DeepSeek-V4.1-Flash, we detect no difference between harnesses. On the first the 90% intervals exclude gaps beyond 2 points, and on the second the harnesses changed order between the two runs.

4. Harness effects on pass rate exist, and they depend on the model. Reviewer 49cb is right that limit-ended trials are part of the harness outcome, so OpenCode's deficit of 8 to 9 points on Qwen3.8-27B pool447 is a harness effect. Two checks narrow its cause. Wall clock does not explain it: on that pool no OpenCode trial timed out, and on hard45, where 12 of 90 did, Claude Code's wall clock removes every timeout and leaves passes at 32 of 89 scored trials against 31 of 89. With the output cap raised from 32k to 131k on the 73 affected tasks (two runs), passes rise from 86 to 99 of 146 (p = 0.076), consistent with truncation losing tasks, short of significance.

The same harness is the highest of the three elsewhere. On DeepSeek-V4.1-Flash hard45, OpenCode passes 27 and 27 against mini-SWE-agent's 21 and 20, a lead of 6 and 7 tasks (14 points, 90% interval 7 to 23). On that model's pool447 the same pair is 1.8 points apart. So "which harness is better" has no answer apart from the model, and a harness gap measured on one model does not carry over to another.

Revision (uploaded, changes in blue). Following Reviewer 49cb's "'Not shown to differ' is supported; 'barely moves' is not", Section 4 and the abstract now state four things. (i) No harness leads consistently across models: over seven models on hard45 each harness is top at least once, and the pooled intervals of all three pairs cover zero. (ii) Where the data can give a bound, the bound is 2 to 5 points. (iii) Harness effects on pass rate exist in particular model and harness pairings, and their direction changes with the model: OpenCode trails by 8 to 9 points on one model and leads by 14 on another. (iv) Cost gaps between harnesses depend on the model, from under 10% to 3 to 4 times. This range replaces "up to 3×".

These four points now rest on three times the submission's trials, and we are confident they stand.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2): Table of All Results

This comment holds the table for our general response (1/2): every pass result in the submission together with the runs added in this response.

All results. hard45 rows give tasks passed out of 45; all other rows give pass rate in %. Each number is one run; bold = added in this response.

Task setModelClaude Codemini-SWE-agentOpenCode
pool447¹Qwen3.6-35B-A3B69.771.5, 70.366.6
Qwen3.8-27B78.279.370.1
DeepSeek-V4.1-Flash²82.7, 84.082.7, 80.982.9, 84.3
Claude Opus 5.5 (391 tasks)³95.193.193.4
hard45¹Qwen3.6-35B-A3B11, 1015, 1311, 11
Qwen3.8-27B19/43, 2017, 1817, 14/44
DeepSeek-V4-Flash21, 232219
GLM-5.3-Flash22, 1920, 1818, 18
HY4-Preview23, 2426, 2421, 24
Claude Opus 536, 35, 35n/an/a
DeepSeek-V4.1-Flash²24, 2321, 2027, 27
Claude Opus 5.536, 37, 363738
SpreadsheetBench Verified⁴DeepSeek-V4.1-Flash88.3, 88.588.8, 87.088.0, 88.8
DABstep⁴DeepSeek-V4.1-Flash74.0, 74.772.0, 76.071.3, 75.3

¹ hard45 is the 45-task hard subset of SWE-bench Verified; pool447 is the other 447 usable tasks, of which 445 can be scored. A hard45 cell shows its denominator when infrastructure failures left fewer than 45 scored tasks. Claude Opus 5 was run in Claude Code only (n/a), so seven models have all three harnesses on hard45.

² The vendor has retired DeepSeek-V4-Flash and its API name now serves V4.1-Flash, which we treat as a separate model. In its runs the harnesses share one soft timeout and keep their default output caps, and no trial reached the timeout or a step limit.

³ Tasks ran in a pre-drawn random order. Our API access to this model was interrupted before the mini-SWE-agent and OpenCode runs finished, so the row gives the 391 tasks that all three harnesses completed. Claude Code completed all 447 and passed 422.

⁴ SpreadsheetBench Verified has 400 tasks; for DABstep we drew 150 of its hard tasks at random. Both are scored deterministically.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.