- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Claw-SWE-Bench, an evaluation benchmark for general-purpose harnesses on SWE-bench-style issue resolution. The benchmark is the union of SWE-bench Multilingual and SWE-bench Verified-Mini.
Critical strengths
The evaluation is exhaustive covering multiple models and harnesses.
Critical weaknesses
W1. The main issue with the paper is the lack of new insights. The benchmark itself is the union of SWE-bench Multilingual and SWE-bench Verified-Mini, with no new tasks, re-validation, or filtering. The actual contribution is just the adapter protocol and the paper should be framed accordingly. The core claim, that agent capability should be reported at the model-harness level because harness choice can move scores by tens of points with the model fixed, is not new. HAL [1], which the authors already cite, provides standardized, cost-accounted evaluation of agents (model + scaffold) across many benchmarks. The authors should state precisely what their protocol adds beyond it rather than citing it in passing.
More importantly, Harness-Bench [2], has essentially the same design. It evaluates 6 harnesses across 8 model backends on 106 sandboxed tasks, 22 of them in software engineering, over 5,194 trajectories, under shared environments and budgets while preserving each harness’s native execution behavior. Two of its findings pre-empt what would otherwise be this paper’s most interesting results: weak models are highly sensitive to the harness while strong models are not, and the top score went to Codex, a specialized coding agent, which is exactly the comparison this paper omits (see W2). The authors need to clarify what Claw-SWE-Bench reveals that Harness-Bench does not.
W2. No comparison against coding harnesses is provided. Claude Code, Codex, OpenHands, etc. are some of the most used coding harnesses and should be used as the baselines to see how well generalized harnesses match up to them.
W3. No error bars are reported. Generally the variance across the SWE-bench family is 2-5% which is greater than some of the margins reported.
W4. There are no failure mode analysis, which could shed light to why the harnesses have different behaviors.
[1] Kapoor, Sayash, et al. "Holistic agent leaderboard: The missing infrastructure for ai agent evaluation." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Yao, Yilun, et al. "Harness-Bench: Measuring harness effects across models in realistic agent workflows." arXiv preprint arXiv:2605.27922 (2026).
Questions for authors
See weaknesses above