acceptodds
Under review as a conference paper at ICLR 2027

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

Abstract

The software engineering capabilities of general-purpose agent harnesses remain underexplored, and existing benchmarks offer limited support for comparing these harnesses under consistent conditions. To address this gap, we introduce Claw-SWE-Bench, a unified benchmark that enables researchers to systematically assess the capabilities and efficiency of general-purpose harnesses on software engineering tasks. The benchmark contains 350 real-world GitHub issue-resolution instances across eight programming languages and 43 repositories and provides a shared adapter protocol to align task inputs, outputs, and execution environments across harnesses. Experiments show that general-purpose harnesses can effectively resolve real-world software issues and that their success rates and resource consumption vary substantially even when the underlying model is held fixed. To lower evaluation costs and support faster debugging and iteration, we also provide Claw-SWE-Bench Lite, an 80-instance subset designed to preserve the key evaluation properties of the full benchmark. We hope this benchmark will help researchers better evaluate and understand the performance of general-purpose harnesses on software engineering tasks and guide the development of more capable and efficient harnesses.

Primary area:
datasets and benchmarks
Keywords:
Harness Evaluation, General-Purpose Agents, Software Engineering Benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_KkwxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Claw-SWE-Bench, an evaluation benchmark for general-purpose harnesses on SWE-bench-style issue resolution. The benchmark is the union of SWE-bench Multilingual and SWE-bench Verified-Mini.

Critical strengths

The evaluation is exhaustive covering multiple models and harnesses.

Critical weaknesses

W1. The main issue with the paper is the lack of new insights. The benchmark itself is the union of SWE-bench Multilingual and SWE-bench Verified-Mini, with no new tasks, re-validation, or filtering. The actual contribution is just the adapter protocol and the paper should be framed accordingly. The core claim, that agent capability should be reported at the model-harness level because harness choice can move scores by tens of points with the model fixed, is not new. HAL [1], which the authors already cite, provides standardized, cost-accounted evaluation of agents (model + scaffold) across many benchmarks. The authors should state precisely what their protocol adds beyond it rather than citing it in passing.

More importantly, Harness-Bench [2], has essentially the same design. It evaluates 6 harnesses across 8 model backends on 106 sandboxed tasks, 22 of them in software engineering, over 5,194 trajectories, under shared environments and budgets while preserving each harness’s native execution behavior. Two of its findings pre-empt what would otherwise be this paper’s most interesting results: weak models are highly sensitive to the harness while strong models are not, and the top score went to Codex, a specialized coding agent, which is exactly the comparison this paper omits (see W2). The authors need to clarify what Claw-SWE-Bench reveals that Harness-Bench does not.

W2. No comparison against coding harnesses is provided. Claude Code, Codex, OpenHands, etc. are some of the most used coding harnesses and should be used as the baselines to see how well generalized harnesses match up to them.

W3. No error bars are reported. Generally the variance across the SWE-bench family is 2-5% which is greater than some of the margins reported.

W4. There are no failure mode analysis, which could shed light to why the harnesses have different behaviors.

[1] Kapoor, Sayash, et al. "Holistic agent leaderboard: The missing infrastructure for ai agent evaluation." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Yao, Yilun, et al. "Harness-Bench: Measuring harness effects across models in realistic agent workflows." arXiv preprint arXiv:2605.27922 (2026).

Questions for authors

See weaknesses above

Reviewer_zuU5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Claw-SWE-Bench evaluates seven general-purpose agent harnesses across three models on 350 existing SWE-bench tasks. It provides a shared evaluation pipeline, removes accessible future-commit history, and selects an 80-task subset for cheaper evaluation.

Critical strengths
  1. Comparing harnesses under a shared setup is useful for choosing between systems. The Git-history cleanup also addresses a concrete evaluation problem: removing accessible fixes changes scores by up to 8 percentage points (Section 6.5).
  2. Lite works well on the six harness–model pairs excluded from calibration. It preserves their ranking and stays within 1.70 points of the full benchmark (Section 6.4).
Critical weaknesses
  1. I am still unsure what new insight to take away from this paper. Work such as SWE-agent (Yang et al., 2024) has already shown that interface design affects agent performance. These experiments quantify the differences more clearly, but most conclusions are about which combinations score higher or cost less. I find that useful as a reference, but not enough to recommend acceptance.
  2. The Meta-Harness experiment leaves me with a similar concern. It starts from GenericAgent, the lowest-scoring harness in Table 2, and the evolved version still has a lower resolved rate than Hermes, although it costs less. Evolution and evaluation also use the same 350 tasks, including when the execution model is changed (Appendix C). I can see the gains for this seed on these tasks. I do not see evidence that they carry over to optimizing stronger harnesses or solving new tasks.
Questions for authors

Beyond the performance and cost rankings, what do you see as the main new finding from these experiments?

Reviewer_KPneOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Claw-SWE-Bench, a benchmark for evaluating OpenClaw-style general-purpose agent harnesses on real GitHub issue-resolution tasks. It combines 300 SWE-bench-Multilingual instances and 50 SWE-bench-Verified-Mini instances, covering 43 repositories and eight language groups. A shared adapter protocol standardizes task inputs, execution environments, patch extraction, and test-based scoring while retaining each harness's internal agent loop.

The main experiment evaluates seven harnesses with three underlying models, repeating each configuration three times. The results show differences in task success and resource consumption even with the model held fixed. The paper also constructs an 80-instance Lite subset using performance, ranking, difficulty, and cost criteria, and validates it on six additional harness–model combinations. A separate diagnostic examines the impact of removing accessible Git history beyond the task's base commit.

Critical strengths

1. A systematic comparison that treats the harness as an explicit variable

The paper holds the underlying model fixed when comparing harnesses and reports both task success and resource consumption. Evaluating seven harnesses with three models and repeating the main runs provides a useful empirical characterization beyond a single model–harness result.

2. Practical attention to evaluation reliability and iteration cost

Collecting patches from repository state avoids dependence on heterogeneous final-response formats. The diagnostic concerning accessible future Git commits highlights an important evaluation-environment issue. The Lite subset also has practical potential for lower-cost debugging, and its evaluation on combinations excluded from subset construction is a useful initial validation.

Critical weaknesses

1. The incremental contribution relative to existing evaluation infrastructure is insufficiently established

The paper presents a harness-agnostic adapter protocol and standardized execution pipeline as central contributions. Given existing agent-evaluation infrastructure such as Harbor, the manuscript needs a concrete account of what prevents an existing platform from supporting the proposed comparison and what nontrivial capability the new protocol adds.

This is not an argument that Harbor and Claw-SWE-Bench are identical: a general evaluation framework and a particular benchmark, subset-selection method, and empirical study are different contributions. However, because the tasks and test-based evaluator are inherited from the SWE-bench family, the paper needs to establish more clearly why the infrastructure warrants a separate contribution and what research value remains beyond integrating additional harnesses.

The adapter comparison in Section 6.1 does not resolve this question. The bare configuration suffers a 69.1% patch-application failure rate, whereas the full adapter exports patches from edited repository files. The large improvement demonstrates the importance of a reliable output contract, but it does not establish an advantage over an existing implementation that already uses file editing and robust patch collection. A capability-level comparison with Harbor, using the version available at submission time, would make the motivation and incremental value more persuasive.

2. The harness study remains largely descriptive rather than explanatory

The main results establish that complete harnesses differ in success rate and cost. Nevertheless, replacing a whole harness jointly changes prompts, tool interfaces, context management, execution policies, and resource usage. The current experiments therefore do not explain which factors account for the differences.

For example, Section 6.2 reports that nanobot's gap to the best harness is much larger with Qwen than with DeepSeek. This suggests an interaction between model and harness, but the paper does not sufficiently examine the tasks or execution behaviours behind it. Similarly, the cost analysis identifies trade-offs without adequately decomposing them into useful exploration, redundant context, unsuccessful tool calls, retries, or other execution behaviours.

A component ablation is not mandatory for every benchmark paper, and the system-level comparison has value on its own. Nevertheless, with much of the infrastructure and task formulation building on existing resources, I would expect a stronger empirical analysis to establish a substantive research contribution. Task-level paired comparisons, failure-stage categorization, representative trajectory analysis, or focused ablations would help turn the performance tables into actionable insights about harness design.

3. Evidence for transfer beyond SWE issue resolution is limited

The experiments cover multiple languages and repositories, but all evaluated tasks follow the SWE-bench issue-resolution setting. The additional Lite validation changes harness–model combinations within that task setting; it does not, by itself, establish whether harness rankings or cost–performance relationships transfer to other task types.

Terminal/CLI tasks, ML experimentation, or data-processing workflows could place different demands on environment interaction, tool use, and long-horizon execution. A small cross-task validation would help distinguish general harness properties from adaptation to a particular repair workflow.

I do not require a software-engineering benchmark to cover all domains, nor do I regard the absence of such experiments as invalidating its in-scope SWE results. My concern is that the current evidence provides a narrower basis for general-purpose harness design guidance than the framing may suggest. This point is secondary to the unclear incremental contribution and limited explanatory analysis; it could also be addressed by explicitly restricting the conclusions to the evaluated issue-resolution setting.

Questions for authors
  1. Differentiation from Harbor: What concrete limitations of Harbor motivated the new adapter protocol? Which required capabilities could not be implemented through existing extension interfaces? Please distinguish the infrastructure contribution from the Lite selection method and empirical findings, and identify the relevant versions available at submission time.

  2. Explanation of harness differences: Can task-level or trajectory-level evidence explain why the leading harnesses perform better, and why relative performance changes across models? What evidence connects these differences to specific tool, context-management, or execution choices rather than to configuration and resource-consumption differences?

  3. Scope and transfer: Is there evidence that the observed rankings and efficiency trade-offs persist on different task types, such as terminal/CLI or ML workflows? If not, which conclusions should be understood as specific to SWE-bench-style issue resolution, rather than general properties of the harnesses?

Reviewer_8NmzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

General-purpose agent harnesses ("claws") such as OpenClaw, Hermes Agent, nanobot and ZeroClaw are widely used, but they have not been compared in a controlled way on coding tasks. The authors combine SWE-bench-Multilingual (300) and SWE-bench-Verified-Mini (50) into a 350-task benchmark, with an adapter that runs each harness in the official image under a fixed prompt and timeout and extracts the patch from the git diff. They also identify and remove a git-history leak in the Multilingual images. Seven harnesses are evaluated with three models, three runs each. With the model fixed, Pass@1 varies by up to 24.3 pp and cost by several times, and Hermes Agent performs best with all three models. The paper also proposes an 80-task Lite subset.

Critical strengths
  • The comparison is well controlled: the model, prompt, image, timeout and patch extraction are fixed and only the harness varies, across 21 pairs with three runs each. The cost numbers are consistent with the token counts and prices (for OpenClaw/GLM I get about $337 vs. $338 reported).
  • The cost results are of practical use. The same model can cost about six times more under one harness than another (GLM: $86 vs. $525) for fairly small differences in Pass@1.
  • The git-history leak (Sec. 6.5) is worth documenting. A similar leak in SWE-bench Verified was reported publicly in 2025 (SWE-bench GitHub issue #465), but the paper shows that it remains in the Multilingual images and measures its effect: removing it lowers Pass@1 by up to 8 pp (Opus 4.7). The earlier report should be cited.
Critical weaknesses
  • The benchmark contribution is limited. There are no new tasks (it is Multilingual plus Verified-Mini), and nothing in the setup tests what distinguishes claws from coding agents, such as memory, skills or multi-session use.
  • No dedicated coding harness (e.g., mini-SWE-agent or OpenHands) is run with the same models, so the claim that claws have "strong coding capabilities" (L79, L480) lacks a point of comparison.
  • There is no variance or significance analysis, even though each configuration is run three times. Hermes's lead over OpenClaw is small (1.9 pp), and it is exactly 6.7 resolved instances under each of the three models (256.0 vs. 249.3, 219.7 vs. 213.0, 286.0 vs. 279.3), which is surprising for independent stochastic runs.
  • The "model held fixed" premise is weakened by claw-specific model settings that are only documented in a README (L740), unpinned OpenRouter routing, and max_turns applying only "where supported".
  • The Lite validation is mostly in-sample, since it is measured on the same 21 columns the subset was selected to match. The reported calibration mean (0.6610) also does not match the 66.4% average in Table 2.
  • With DeepSeek, all harnesses score between 71.5 and 81.7% on older public issues, so headroom is limited, and contamination is not discussed.
Questions for authors
  1. What is the standard deviation across the three runs, and what are the paired confidence intervals for the within-model differences? Does Hermes's 1.9 pp lead hold up, and why is it exactly 6.7 instances under all three models?
  2. What exactly differs in the "claw-specific model settings"? Is persistent state (memory, skills, workspace files) cleared between instances?
  3. Could you add at least one dedicated coding harness with the same three models?
  4. For Lite: which runs were used for calibration, why is the mean 0.6610 rather than 66.4%, and how does the subset compare with random stratified subsets on the held-out pairs?
  5. How many trajectories before the cleanup actually accessed future commits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.