acceptodds
Under review as a conference paper at ICLR 2027

How good is your harness? Statistical evaluation of coding harnesses

Abstract

AI agent benchmarks conflate the underlying language model with the harness that wraps the model in tools, prompts, and control flow. We develop a simple statistical method to disentangle the effects of the LLM and its harness on the agent's score; i.e., to attribute the variation in agents' scores to their LLMs and harnesses. We use the method to evaluate harnesses and LLMs on Terminal-Bench 2.0, and our results show that 1. the harness can matter as much as an LLM upgrade: the range of harness effects is comparable to the gain from Claude Opus 4.1 to Claude Opus 4.6. 2. the harness effects can be heterogeneous; i.e., some harnesses work better with some LLMs than with others. 3. the effects of individual harness features can also vary across LLMs. Our results confirm and, more importantly, quantify practitioners' intuition on the importance of the harness and validate efforts to tailor harnesses to specific applications.

Primary area:
datasets and benchmarks
Keywords:
agentic benchmarking, harness evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XpAUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

In an observational analysis of the Terminal-Bench 2.0 leaderboard, the authors use an additive generalized linear model (GLM) to estimate harness effects while adjusting for LLM identity, and LLM effects while adjusting for harness identity. The estimated gap between the highest- and lowest-ranked harnesses is similar to the gap between Claude Opus 4.1 and Claude Opus 4.6 on the log-odds scale. To test whether harness effects vary across LLMs, the authors analyze nine LLMs evaluated with Mini-SWE-Agent, OpenHands, and Terminus 2. They find that OpenHands' advantage over Mini-SWE-Agent varies across these LLMs.

To examine individual harness features, the authors study little-coder with three LLMs: Gemma-4-26B-A4B (Gemma), K2-Horizon-MoVA-36B-A4B (K2), and Qwen3.6-35B-A3B (Qwen). For each LLM, they randomly select which features to enable and test the resulting configurations on the same Terminal-Bench 2.0 tasks. Infrastructure (INFRA) is always enabled. The analysis excludes configurations without shell session (SS), which provides the shell tool needed to run the tasks. Both the additive model and KernelSHAP find that quality monitoring contributes positively on average across the three LLMs.

Critical strengths

S1: The comparison with LLM upgrades helps interpret the harness effects. Expressing both on the same log-odds scale gives the reader a useful reference for judging the magnitude of harness differences. The authors appropriately distinguish these observational comparisons from causal estimates. The authors also compare OpenHands and Terminus 2 with Mini-SWE-Agent using nine LLMs evaluated with all three harnesses. On this scale, they reject the hypothesis that the OpenHands–Mini-SWE-Agent difference is constant across LLMs, but do not reject the hypothesis of a constant Terminus 2–Mini-SWE-Agent difference.

In little-coder, the quality monitor detects failures such as empty responses and repeated loops and prompts the agent to retry. Across Gemma, K2, and Qwen on Terminal-Bench 2.0, with INFRA and SS enabled, the authors estimate the largest average contribution for quality monitoring in both the additive feature model and KernelSHAP. KernelSHAP estimates feature contributions without imposing additivity, allowing a feature's contribution to depend on which other features are enabled. It gives wider confidence intervals here, but quality monitoring is the only feature with a significantly positive average contribution under both methods. This provides a check on the additive model using the same observations, rather than an independent replication.

S2: The robustness checks support both the harness comparison and the average benefit of quality monitoring. For the Terminal-Bench 2.0 leaderboard, the authors add task fixed effects to their additive harness–LLM model to account for task difficulty. This increases the coefficients but leaves the harness and LLM rankings essentially unchanged. The estimated ForgeCode–Mini-SWE-Agent difference remains comparable to the estimated Claude Opus 4.6–Claude Sonnet 4.5 difference on the log-odds scale. For their binomial GLMs, the authors compare model-based standard errors with trial-row HC0 sandwich standard errors. They find that the HC standard errors are only slightly different and yield qualitatively identical results: the intervals exclude zero for the same coefficients and contrasts under both methods. Both methods still treat trials as independent.

For the little-coder configurations on Terminal-Bench 2.0 with INFRA and SS enabled, the authors also provide confidence intervals for the KernelSHAP feature contributions under an alternative uncertainty estimate. This calculation counts some noise in the observed configuration scores twice. The alternative calculation still supports a positive average contribution of quality monitoring across Gemma, K2, and Qwen: the interval for that average remains entirely above zero. The wide interval leaves the size of that benefit uncertain.

Critical weaknesses

1. The standard errors treat correlated outcomes as independent. Terminal-Bench 2.0 evaluates coding agents on 89 terminal-based tasks, with 5 trials per task, and Each GLM is fitted to trial rows: one binary outcome Yi per harness–LLM–task–repeat trial. The model-based errors assume that the working GLM is correctly specified, yet The additive model in Equation 2.1 has no task term, so shared task difficulty enters its residuals. The paper states this itself: Outcomes on the same task may share a difficulty that the GLMs omit. The sensitivity check does not address it, because Like the model-based errors, the sandwich treats trials as independent, The two estimators therefore differ only through the entry-level lack of fit, and Independence of trials alone therefore does not make HC0 consistent for re-execution variance under mean misspecification. The refit with task fixed effects changes the coefficients (ForgeCode − Mini-SWE-Agent goes from 0.890 to 1.594), but Both fits use model-based standard errors. So the conclusion that No coefficient or contrast changes its significance call, so every bold entry in Tables 1–4 is unchanged compares two estimators that share the same independence assumption. Standard errors clustered by task, or by entry and task, are needed. The per-category fits are affected most: File operations (5) has five tasks, and ForgeCode's 2.121 ± 0.494 is bold.

2. The leaderboard entries are self-selected. The paper states that The fitted effects are adjusted comparisons from an observational leaderboard, not causal estimates. Its headline findings still say that the harness can matter as much as an LLM upgrade and that a better harness can pay off as much as upgrading to a newer flagship LLM, which read as causal. The paper does not discuss how the 118 entries of The leaderboard snapshot we collected were selected by their submitters. Section 5 describes how ForgeCode's model-specific fixes—schema field ordering, flatter schemas, explicit truncation notices, and enforced reviewer-mode verification—brought both GPT-5.4 and Opus 4.6 to 81.8% on Terminal-Bench 2.0. Part of ForgeCode's coefficient may therefore reflect fixes made for this benchmark, not a harness effect in general.

3. The central comparison sets a range against a single contrast. Our central empirical result comes from comparing the harness effects with familiar LLM upgrades: the harness effects span 1.12 logits, from Gemini CLI (−0.20) to Ante (0.93), comparable to recent progress within a flagship line: 1.01 logits from Claude Opus 4.1 to Claude Opus 4.6. The span is the gap between the two most extreme of 28 estimates, while the Opus contrast is one difference. The intervals are reported without adjustment for multiple comparisons or selection of the largest effects, and the span is exactly such a selection. Its top end is Ante, and Ante has the largest estimated harness effect, but it is observed with only one LLM, so its coefficient may also reflect a pairing-specific interaction. This is not a single case, because many harnesses are evaluated on only one or two LLMs. Shrinkage of harness effects, or a span restricted to harnesses with more than one entry, would make the comparison fair.

4. Harness and LLM levels may hide different configurations. After normalizing duplicate LLM names, this corresponds to 39 LLM levels, but the paper does not say whether entries with different reasoning settings or model versions were merged into one level. Leaderboard harness versions are not reported either, although the paper itself notes for little-coder that the two Terminal-Bench entries also differ in harness version. Part of a harness effect may then be an effect of model settings or of harness version.

5. The interaction finding rests on one harness. The abstract states that the harness effects can be heterogeneous; i.e., some harnesses work better with some LLMs than with others. In the 3 × 9 block, The departures come from OpenHands (G² = 63.7, 8 df, p < 10⁻¹⁰) rather than Terminus 2 (G² = 5.5, 8 df, p = 0.71), and for Terminus 2 all of its non-reference intervals include zero. The evidence for heterogeneity therefore comes from one harness. The interactions are negative for Claude Haiku 4.5 and Gemini 2.5 Pro (−0.95 and −0.58), and the paper offers only a possibility: differences in how LLMs handle tool calls interact with OpenHands' action interface, though the success-rate analysis does not identify this mechanism. Error and timeout rates per cell would show whether these values reflect harness–LLM fit or failed runs.

6. The interaction estimates depend on the scale and on the reference. Under our reference coding, the interaction is a difference-in-differences in log-odds, computed only with the logit link, and the paper does not check whether the OpenHands pattern holds on another scale. The paper states that The references are only a coding convention: changing them shifts every harness (or LLM) coefficient by the same constant, leaving the rankings and the differences between coefficients unchanged. This holds for the additive model. In Table 3, however, each γij measures how the harness contrast changes relative to the reference LLM, rather than a residual from the separately fitted additive model, and interaction terms involving either reference level are zero by construction. The reported pattern, positive Relative to Claude Sonnet 4.5 for the GPT-5 models, therefore depends on choosing Claude Sonnet 4.5. Only the likelihood-ratio test does not depend on which harness and LLM serve as references.

7. The KernelSHAP analysis adds complexity without new conclusions. The constraint enforces Shapley efficiency: the nine feature attributions sum to the empirical log-odds difference between the full and baseline configurations, and each configuration is evaluated with one recorded attempt per task and a binary success outcome. For Qwen, the baseline solves 26/89 (29.2%) tasks and the full configuration 21/89 (23.6%), so its nine attributions must sum to −0.290. The paper itself says that KernelSHAP allows feature interactions at the cost of wider intervals. Most of that width is noise in the coalition values, amplified by the endpoint constraint and the concentrated kernel weights. Its variance construction is our approximation, its degrees of freedom are calibration approximations, and clipping guarantees neither an upper bound on the true variance nor nominal finite-sample coverage. The paper concludes that The additive GLM gives the more precise summary, and that the sign changes of EV, KS and TB remain inconclusive. Since the feature GLM excludes interactions among harness features, adding pairwise interactions to it and testing them with a likelihood-ratio test would check additivity more directly.

8. The method itself is standard. The abstract says We develop a simple statistical method to disentangle the effects of the LLM and its harness on the agent's score. The paper also says We use standard generalized linear models (McCullagh & Nelder, 1989) to decompose variation in agent scores into harness and LLM effects and We fit Equation 2.1 to data with off-the-shelf logistic regression software. The Shapley reading adds no new quantity: In our two-attribute reference game these first-order components are exactly the Shapley contributions of harness and LLM identity on the link scale, and Hence, in the additive model, the harness and LLM coefficients are simultaneously the first-order functional components and the ordinary BShap attributions relative to the reference pair. The contribution lies in the application to Terminal-Bench 2.0 and in the collected data. The paper should either present itself as an empirical study or show what the approach gives beyond a standard GLM for a design that is sparse and unbalanced.

Questions for authors
  1. Outcomes on the same task may share a difficulty that the GLMs omit. Do the bold entries in Tables 1–4 remain bold with standard errors clustered by task, or by entry and task? How many of the bold coefficients in Table 2 remain?

  2. The fitted effects are adjusted comparisons from an observational leaderboard. Do you know how the entries in The leaderboard snapshot we collected were produced, in particular whether submitters evaluated several pairings and reported only some of them?

  3. Ante has the largest estimated harness effect, but it is observed with only one LLM. What is the span of harness effects over harnesses with more than one entry only? Which LLMs do ForgeCode's two entries use?

  4. After normalizing duplicate LLM names, this corresponds to 39 LLM levels. Were entries with different reasoning settings or model versions merged into one LLM level? Which harness version does each leaderboard entry use?

  5. The OpenHands interactions are negative for Claude Haiku 4.5 and Gemini 2.5 Pro (−0.95 and −0.58). What are the error and timeout rates in these cells, and does the interaction remain after runs that failed for infrastructure reasons are excluded?

  6. The departures come from OpenHands. Does this remain under another link function, for example probit or identity? How does Table 3 change with another reference LLM, since interaction terms involving either reference level are zero by construction?

  7. KernelSHAP is applied To estimate Shapley attributions without imposing additivity. What does a likelihood-ratio test of pairwise feature interactions in the feature GLM show, and what does KernelSHAP add if The additive GLM gives the more precise summary?

  8. The paper uses standard generalized linear models fitted with off-the-shelf logistic regression software. Which part of the simple statistical method do you consider new, beyond its application to Terminal-Bench 2.0?

3 comments

In response to the Program Chairs’ concern about AI use, I used Grammarly to improve the readability of my review. I identified the paper’s strengths and weaknesses myself. I did not use any AI tools to identify those strengths or weaknesses, form my overall assessment, or determine my score.

AuthorsOpenReview ↗

Response to Reviewer XpAU (1/2)

We thank Reviewer XpAU for the questions and requests for clarification.

W1/Q1: Clustered SEs. Each trial is executed independently, so evaluating multiple agents on the same task does not itself imply dependence between trial outcomes. Moreover, Appendix C already accounts for shared task difficulty through task fixed effects. The resulting harness and LLM rankings remain essentially unchanged (Spearman 0.999/1.000), as do all significance conclusions. HC0 sandwich SEs also yield nearly identical results (median SE ratio 0.999–1.001). As an additional robustness check, we computed task-clustered SEs. All previously significant entries in Tables 1, 3, and 4 remain significant. Clustering by entry is not needed beyond this: all trials of an entry share the same harness–LLM design row, so beyond shared task difficulty they can be correlated only through entry-level lack of fit of the mean model. The 3 × 9 interaction block is saturated and has none, and for the additive model the HC0 SEs, which grow with entry-level lack of fit, are nearly identical to the model-based SEs. For Table 2, most categories contain only 5–9 tasks, too few clusters for reliable cluster-robust inference.

W2/Q2: Entry selection. The leaderboard records only submitted entries, so we cannot determine whether developers evaluated additional pairings that were not reported. We evaluate the submitted harness configurations, including their prompts and tuning, and use the GLM to adjust comparisons for LLM identity. These remain observational comparisons among submitted entries; submission selection may affect the fitted differences. Section 4's controlled experiment is free of this selection: we fixed the three LLMs in advance, randomly sampled the little-coder configurations, and ran and report every configuration on every task.

W3/Q3: Harness span and sparse pairings. Our comparison with an LLM upgrade is intended to provide an intuitive reference for interpreting the magnitude of the estimated harness span, rather than to test the equality of the two contrasts. Both are measured on the same log-odds scale. Restricting to the 17 harnesses evaluated with more than one LLM yields a span of 1.09 logits (Gemini CLI to ForgeCode), still comparable to the Claude Opus 4.1 → 4.6 upgrade (1.01 logits). ForgeCode's two entries use Claude Opus 4.6 and Gemini 3.1 Pro. Thus, the harness span remains comparable to the LLM upgrade even after excluding harnesses evaluated with only one LLM.

W4/Q4: LLM and harness configurations. The leaderboard specifies a common evaluation protocol requiring repeated trials under the same configuration. We did not merge distinct LLM versions; only three naming variants were standardized. Reasoning settings are not separately recorded in the leaderboard, and harness versions are available for only some entries. All OpenHands entries in the 3 × 9 interaction analysis use version 0.60.0.

W5/Q5: Error and timeout rates. The leaderboard reports each trial's outcome only as pass or fail, without distinguishing infrastructure failures, timeouts, or unsuccessful task attempts. Therefore, these rates cannot be separately computed, and infrastructure-related failures cannot be identified and excluded from the analysis.

W6/Q6: Link function and reference coding. The OpenHands interaction remains significant under logit, probit, and identity links (LRT G² = 63.7/63.7/63.6, p < 10⁻¹⁰), while Terminus 2 remains non-significant (G² = 5.5–5.6, p ≥ 0.69). On the probability scale, OpenHands' relative advantage is 11.6 ± 2.1 percentage points larger for GPT-5 models. We agree that individual interaction coefficients depend on the reference LLM, but the differences in harness advantages across LLMs and the overall interaction tests are reference-invariant. Thus, our main conclusions do not depend on the choice of link function or reference coding.

W7/Q7: Feature interactions and KernelSHAP. Adding all 45 pairwise feature interactions to the GLM yields no significant improvement (joint LRT G² = 46.3 on 135 df, p > 0.99). KernelSHAP adds a check that does not impose additivity: it targets Shapley values, which remain interpretable under feature interactions. It agrees with the GLM in sign for quality monitoring and write protection, but its intervals are wider because the Shapley kernel concentrates leverage on a few configurations; these configurations are not outliers. Dropping the endpoint constraints gives the same picture (quality monitoring: 0.077 [−0.029, 0.182] constrained, 0.084 [−0.004, 0.171] unconstrained), so the conclusion does not depend on the constraint; we keep it because it enforces Shapley efficiency. We therefore treat the GLM as the primary analysis and KernelSHAP as a check on it. Please see Appendix D.4 of the revised paper for the full comparison of the two analyses.

AuthorsOpenReview ↗

Response to Reviewer XpAU (2/2)

W8/Q8: Contribution. We respectfully disagree that the use of established statistical methods limits the novelty or informativeness of our contribution. Our central contributions are important new empirical findings about coding harnesses, not new statistical estimators. While practitioners have long recognized that harnesses matter, the magnitude of their effects, their dependence on the underlying LLM, and the contributions of individual harness features have not been systematically quantified.

Our study establishes three concrete findings:

  1. Harness choice can be as consequential as an LLM upgrade: the range of harness effects is comparable to the Claude Opus 4.1 → Opus 4.6 upgrade (1.12 vs. 1.01 logits).
  2. Harness effectiveness depends on the underlying LLM: OpenHands' relative advantage is 0.70 ± 0.13 logits larger for GPT-5 models than for Claude and Gemini models.
  3. Most harness features have no detectable effect: of ten feature groups, only quality monitoring improves little-coder's average success on Terminal-Bench (0.133 ± 0.030), and write protection even reduces it (−0.066 ± 0.030).

These findings have direct practical implications: harness choice deserves as much attention as LLM choice, harnesses should be tailored to the LLM, and harness features should be measured on the target LLMs and tasks rather than added by default. We revised the third finding in the abstract and introduction to reflect the updated experiment.

Beyond these empirical findings, we carefully quantify statistical uncertainty, examine robustness, and establish a connection between our GLM decomposition and Shapley attribution. We also explored several Shapley estimation approaches and selected the standard formulation as the most suitable for our harness evaluation setting.

If the reviewer knows of prior work that has already addressed these empirical questions (e.g., studies quantifying how much a coding harness contributes to agent performance relative to the LLM), we would appreciate relevant references.

Reviewer_oEFuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper fits an additive logit GLM to the Terminal-Bench 2.0 leaderboard (105 harness-LLM pairs, 28 harnesses, 26 LLMs; reference Mini-SWE-Agent + Sonnet 4.5) to separate harness effects from LLM effects. Harness effects span 1.12 logits versus 4.10 for LLMs, and the best harness effect (ForgeCode, +0.89) is comparable to a flagship model upgrade (Opus 4.6, +0.84). A saturated 3 x 9 block (135 rows, 11,941 outcomes) rejects additivity (G^2 = 91.2 on 16 df); for example, OpenHands helps GPT models by 0.70 +/- 0.13 more than others. A feature ablation with little-coder across 171 profiles on Gemma-4-26B, K2-Horizon and Qwen3.6-35B finds only quality monitoring significant (v = 0.186), with five features inert under the shell-only interface; KernelSHAP agrees.

Critical strengths
  • Useful question and clear message. Quantifying harness versus model contributions is practically important, and the 'harness gap comparable to a model upgrade' finding is memorable.
  • Interaction analysis. Testing additivity and identifying harness-LLM interactions is a valuable addition beyond leaderboard ranking.
  • Controlled complement. The feature ablation adds an experimental component to the observational analysis. Its causal interpretation is restricted to the tested implementation, interface and operationally active features.
  • Clear writing. The statistical setup is well explained.
Critical weaknesses
  • Limited methodological novelty. The approach is a standard GLM/ANOVA decomposition.
  • Central concern: limited operational scope of feature interventions. Five feature groups are largely inert under the shell-only interface. Their null effects cannot establish that those features are generally unhelpful. This weakens the controlled experiment as evidence for broad harness-design conclusions. -Single benchmark. All conclusions come from Terminal-Bench 2.0; inconsistencies in leaderboard trial counts are not fully addressed.
  • Observational, non-causal effects. Leaderboard entries are self-submitted and differ in more than the harness (tuning effort, timeouts, prompts, trial counts), so 'harness effects' absorb these differences; selection bias is likely.
  • Sparse design. Many harnesses appear with only one or two LLMs (Ante has a single entry), so their effects are weakly identified.
  • Dependence assumptions remain a concern. Appendix C reports sandwich standard errors and task-fixed-effect checks; these are meaningful robustness analyses, not missing experiments. The remaining issue is whether uncertainty adequately accounts for dependence among repeated task-trial outcomes and shared tasks across submissions.
  • Average effects require qualified interpretation. The interaction test rejects strict additivity. This does not make an additive first-order summary invalid, but it limits interpretation of a universal harness ranking. The paper should emphasize model-conditional effects and ranking stability rather than exchangeability of harnesses.
Questions for authors
  1. Beyond Appendix C's sandwich and task-fixed-effect checks, how sensitive are uncertainty estimates to task-level clustering or a dependence-aware resampling scheme?
  2. Can you control for submission-level covariates (trial count, timeouts, submitter) or restrict to entries run under a common protocol?
  3. How stable are harness rankings if harnesses with fewer than three LLM pairings are excluded?
  4. Given the rejection of additivity, how should practitioners interpret the additive harness ranking?
  5. Does the analysis replicate on another benchmark (e.g., SWE-bench Verified leaderboard)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer oEFu (1/2)

We thank Reviewer oEFu for the careful assessment of our statistical analysis and for recognizing the value of our interaction analysis and robustness checks.

Q1: Clustering and resampling. Appendix C already addresses shared task difficulty through task fixed effects and examines sensitivity to the variance specification through HC0 sandwich SEs. Rankings are essentially unchanged (Spearman 0.999 for harnesses, 1.000 for LLMs), and no significance conclusion changes. In particular, the task fixed-effects model directly addresses the reviewer's concern that outcomes on the same task share a common difficulty, by explicitly accounting for task-specific differences rather than leaving them as unexplained variation.

For the overall additive model in Section 2, we additionally examined task-clustered SEs (CR1, t(88)) and a task bootstrap (2,000 replicates), allowing arbitrary dependence among outcomes on the same task. All coefficients displayed in Table 1 remain significant under both. Of the 42 originally significant harness and LLM coefficients in this model, 40 and 39 remain significant, respectively.

Q2: Submission-level covariates. The leaderboard's submission instructions require each harness–LLM configuration to be evaluated at least five times on every task, with timeout multiplier 1.0 and no timeout or resource overrides. We evaluate each harness in its submitted configuration, including its prompts and tuning choices.

Q3: At least three pairings. Requiring at least three LLM pairings leaves 11 harnesses (81 entries). The retained ranking is unchanged (Spearman 1.00), but the range shrinks to 0.63 ± 0.10 logits (Gemini CLI to Droid), as the remaining harnesses are mostly general-purpose ones that lie close together. This is still comparable to LLM upgrades in the same fit: GPT-5 → GPT-5.2 (0.63 ± 0.08), GPT-5-Mini → GPT-5 (0.63 ± 0.07), Claude Opus 4.1 → Opus 4.5 (0.67 ± 0.07), and Claude Haiku 4.5 → Sonnet 4.5 (0.68 ± 0.06). With at least two pairings, the range is 1.09, still comparable to Opus 4.1 → 4.6 (1.01). The reported SEs already reflect the uncertainty from sparse pairings.

Q4: Interpreting the additive ranking. An additive harness coefficient should be read as the harness's average advantage, adjusted for the LLM, over the LLMs it was submitted with, not as its advantage with every LLM. In the 3 × 9 block, the departure from additivity comes from one harness: OpenHands' advantage is 0.70 ± 0.13 logits larger for GPT-5 models than for Claude and Gemini models, while Terminus 2 shows no significant interaction (G² = 5.5, 8 df, p = 0.71). Practitioners can therefore use the additive ranking for a broad comparison and, when choosing a harness for a particular LLM, consult the model-specific comparisons where available (Section 5). Beyond ranking individual harnesses, the additive analysis shows that, on average over LLMs, harness effects are as large as an LLM upgrade, which underscores for practitioners the importance of harness development.

Q5: Another benchmark. Yes. On SWE-bench Verified (30 entries, 14 harnesses, 9 LLMs), Eq. 2.1 gives a harness span of 1.16 ± 0.18 logits, against 1.37 ± 0.13 for Claude 3.5 Sonnet → Sonnet 4. OpenHands vs. mini-SWE-agent (0.39 ± 0.07) matches Opus 4 → 4.5 (0.37 ± 0.10). Its limits: 8 of 14 harnesses have one LLM, and no fully observed block comparable to the 3 × 9 Terminal-Bench block exists; the largest is 2 harnesses × 4 LLMs (OpenHands and mini-SWE-agent with Claude Opus 4.5, Claude Sonnet 4, GPT-5, and Kimi K2). In this block, the harness–LLM interaction is significant (likelihood-ratio G² = 20.5, 3 df, p < 0.001), consistent with the OpenHands interaction in Section 3.

AuthorsOpenReview ↗

Response to Reviewer oEFu (2/2)

We would also like to address several points raised in the reviewer's assessment of our paper's weaknesses.

W: Methodological novelty. Our central contributions are important new empirical findings about coding harnesses, not new statistical estimators. We use established statistical methods to answer important questions about harness design:

  1. Harness choice can be as consequential as an LLM upgrade: the range of harness effects is comparable to the Claude Opus 4.1 → Opus 4.6 upgrade (1.12 vs. 1.01 logits).
  2. Harness effectiveness depends on the underlying LLM: OpenHands' relative advantage is 0.70 ± 0.13 logits larger for GPT-5 models than for Claude and Gemini models.
  3. Most harness features have no detectable effect: of ten feature groups, only quality monitoring improves little-coder's average success on Terminal-Bench (0.133 ± 0.030), and write protection even reduces it (−0.066 ± 0.030).

These findings have direct practical implications: harness choice deserves as much attention as LLM choice, harnesses should be tailored to the LLM, and harness features should be measured on the target LLMs and tasks rather than added by default. We revised the third finding in the abstract and introduction to reflect the updated experiment.

Beyond these empirical findings, we carefully quantify statistical uncertainty, examine robustness, and establish a connection between our GLM decomposition and Shapley attribution. We also explored several Shapley estimation methods and selected the standard approach as the most suitable for our harness evaluation setting. Our choice of established methods reflects their suitability for the research questions, rather than a lack of methodological care.

W: Operational scope of the feature interventions. We agree that null effects from inert features would not establish that those features are unhelpful. We therefore modified little-coder's Terminal-Bench adapter so that every feature group can operate. For example, the browser tools are called in 154 trials with BR enabled and in none with it disabled. We then repeated the ablation on all three LLMs with all ten groups as factors, including shell session, which is no longer the only shell channel. In the GLM, quality monitoring remains the only feature with a significant positive contribution (0.133 ± 0.030), and write protection, now operational, shows a small negative contribution (−0.066 ± 0.030). The other previously limited groups show no detectable effect on this benchmark (Section 4 and Table 4 of the revised paper). Under KernelSHAP, Q and W have the same signs as in the GLM, with intervals that include zero.

W: Observational, non-causal effects. We agree. Sections 2 and 3 are observational: they compare harnesses and LLMs as submitted, with prompts and tuning choices treated as part of each harness, and self-selection may affect which pairings are submitted, including whether submitters report their best-tuned configurations. We therefore do not claim causal effects from the leaderboard analysis: its estimates are LLM-adjusted comparisons among submitted entries, not the gains from switching harnesses.

Section 4 complements the observational analysis with a controlled experiment that is free of this selection. To estimate harness feature effects, we fixed the three LLMs in advance, randomly sampled the little-coder configurations, ran every configuration on every task ourselves, and report all outcomes; no LLM, configuration, or run was selected on its results.

Reviewer_AqzyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the effects of coding harnesses and LLMs on agent performance. It uses logistic regression to estimate their individual and interaction effects from the Terminal-Bench 2.0 leaderboard. The authors also evaluate different features of the little-coder harness across three LLMs, using regression and KernelSHAP to measure their contributions. The results show that harness choice matters, its effect varies across LLMs, and quality monitoring improves performance.

Critical strengths
  1. Relevant problem. The paper studies how coding harnesses affect agent performance, which is often overlooked in LLM evaluations. Separating harness and model effects can help make benchmark comparisons more meaningful.
  2. Useful statistical analysis. The paper studies harness and tests individual harness features. The task fixed-effect analysis supports the main findings, while both regression and KernelSHAP show a positive average effect from quality monitoring.
Critical weaknesses
  1. Limited novelty. The method mainly uses standard logistic regression and KernelSHAP to estimate harness and LLM effects. These are well-established tools, and the technical contribution beyond applying them to agent evaluation seems limited.
  2. Sparse leaderboard data. The analysis covers 105 harness–LLM pairs across 28 harnesses and 26 LLMs. Some top-ranked harnesses are tested with only one or two LLMs. Given the strong interaction effects in Sec. 3, the rankings may depend heavily on which models were tested. This also makes the comparison between harness improvements and model upgrades less convincing.
  3. Limited feature evaluation. The ablations use only little-coder on Terminal-Bench 2.0. Five of the nine optional features have limited use under the shell-only interface. The results therefore tell us little about these features beyond quality monitoring.
  4. Writing and presentation. The paper repeats similar findings across the introduction, results, and discussion. Some conclusions are also stated too broadly. In particular, the difference between estimated effects from leaderboard data and actual gains from changing a harness is not always clear.
Questions for authors
  1. What is the main contribution beyond applying existing statistical methods to agent evaluation?
  2. How stable are the harness rankings when excluding harnesses tested with only one or two LLMs? Have the authors tried a leave-one-LLM-out analysis?
  3. Can the authors test the harness features in a setting where all of them can work as intended?
  4. Can the authors clarify which conclusions are supported by the observational data and which would require controlled experiments?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Aqzy (1/2)

We thank Reviewer Aqzy for the four questions. We clarify our contribution and address each point below.

W1/Q1: Contribution. We respectfully disagree that the use of established statistical methods limits the novelty of our contribution. Our central contributions are important new empirical findings about coding harnesses, not new statistical estimators. While practitioners have long recognized that harnesses matter, the magnitude of their effects, their dependence on the underlying LLM, and the contributions of individual harness features have not been systematically quantified. Our study establishes three concrete findings:

  1. Harness choice can be as consequential as an LLM upgrade: the range of harness effects is comparable to the Claude Opus 4.1 → Opus 4.6 upgrade (1.12 vs. 1.01 logits).

  2. Harness effectiveness depends on the underlying LLM: OpenHands' relative advantage is 0.70 ± 0.13 logits larger for GPT-5 models than for Claude and Gemini models.

  3. In our controlled feature evaluation, most harness features have no detectable effect: of ten feature groups, only quality monitoring improves little-coder's average success on Terminal-Bench (0.133 ± 0.030), and write protection even reduces it (−0.066 ± 0.030).

These findings have direct practical implications: harness choice deserves as much attention as LLM choice, harnesses should be tailored to the LLM, and harness features should be measured on the target LLMs and tasks rather than added by default. We revised the third finding in the abstract and introduction to reflect the updated experiment.

Beyond these empirical findings, we carefully quantify statistical uncertainty, examine robustness, and establish a connection between our GLM decomposition and Shapley attribution. We also explored several Shapley estimation methods and selected the standard approach as the most suitable for our harness evaluation setting. Our choice of established methods reflects their suitability for the research questions, rather than a lack of methodological care.

If the reviewer knows of prior work that has already addressed these empirical questions (e.g., studies quantifying how much a coding harness contributes to agent performance relative to the LLM), we would be grateful for the references.

W2/Q2: Stability. The harness rankings remain highly stable under both checks.

  • Excluding sparse harnesses. Requiring at least two LLMs per harness leaves 17 harnesses (93 entries); at least three leaves 11 (81). The rankings of retained harnesses are unchanged (Spearman 1.00), with median absolute changes in harness coefficients of at most 0.01 logits. With two, the harness span is 1.09 (Gemini CLI −0.20 to ForgeCode +0.89), still comparable to Opus 4.1 → 4.6 (1.01). With three, the span decreases to 0.63 ± 0.10 (Gemini CLI to Droid), as the restriction retains a smaller, more homogeneous set of harnesses. Importantly, even this restricted span is comparable to several within-line LLM upgrades in the same fit: GPT-5 → GPT-5.2 (0.63 ± 0.08), GPT-5-Mini → GPT-5 (0.63 ± 0.07), Opus 4.1 → Opus 4.5 (0.67 ± 0.07), and Haiku 4.5 → Sonnet 4.5 (0.68 ± 0.06).
  • Leave-one-LLM-out. Across the 25 refits in which the remaining design is identified, harness rankings remain highly correlated with the full fit (median Spearman 0.999; minimum 0.979). Removing Opus 4.6 disconnects ForgeCode, Terminus-KIRA, TongAgents, and Gemini 3.1 Pro from the rest of the design, so their coefficients are not identified relative to the reference; we exclude this refit. LLM rankings are even more stable (Spearman ≥ 0.998). The harness span remains within 1.05–1.24 logits, while the Opus 4.1 → 4.6 contrast remains within 0.94–1.04 wherever both models are retained.
AuthorsOpenReview ↗

Response to Reviewer Aqzy (2/2)

W3/Q3: All features working. Yes. We modified little-coder's Terminal-Bench adapter so that every feature group can operate. The agent now has little-coder's native file tools (read, write, edit, grep, bash) and every extension tool, including the browser tools, alongside the shell-session tools. Because the shell-session tools are no longer the only way to run commands, shell session (SS) can now be ablated as well, and we include it as a tenth factor. Without SS, the agent runs commands through the native bash tool (in 13,474 of the 14,503 trials with SS disabled) and solves 26.9% of tasks, compared with 27.9% with SS enabled. For each of Gemma, K2 and Qwen, we ran 100 randomly sampled configurations plus the all-on and all-off configurations, each once on the 89 tasks of Terminal-Bench 2.1, which keeps the task names of Terminal-Bench 2.0 (27,234 task outcomes).

The features are now exercised. For example, the browser tools are called in 154 trials with BR enabled and in none with BR disabled. The file tools that W and CP act on are used in half of all trials.

Quality monitoring (Q) remains the only feature with a significant positive contribution in the GLM (0.133 ± 0.030). With file tools now available, write protection (W) shows a small negative contribution (−0.066 ± 0.030); the other eight groups show no detectable effect (Table R1). Under KernelSHAP, the attributions of Q (+0.077) and W (−0.053) have the same signs as in the GLM, but their intervals are wider (Q: [−0.029, 0.182]) and include zero. For every feature, the GLM estimate lies inside the KernelSHAP interval. The revised paper reports these results in Section 4 (Tables 4 and 5), with the updated experimental setup in Appendix B.2 and the comparison of the two analyses in Appendix D.4.

Table R1. GLM feature contributions averaged over the three LLMs with all features operational (logits, estimate ± SE). Bold entries are significant at the 5% level.

FeatureEstimate
Q (quality monitoring)+0.133 ± 0.030
W (write protection)−0.066 ± 0.030
OP (output parsing)−0.013 ± 0.029
KS (knowledge and skill injection)+0.003 ± 0.030
TB (thinking budget)−0.033 ± 0.030
CP (checkpointing)−0.012 ± 0.030
SS (shell session)+0.021 ± 0.029
BR (browser)+0.047 ± 0.030
ET (extra tools)−0.004 ± 0.029
EV (evidence management)+0.017 ± 0.030

W4/Q4: Observational vs. controlled.

  • Supported by the observational data (Sections 2 and 3): how much harnesses differ in performance among the submitted entries, adjusted for the LLM, and that these differences vary across LLMs in the 3 × 9 block.
  • Would require controlled experiments: the gain from switching to a given harness, and the mechanism behind the interactions. Section 4 is such an experiment for little-coder's features, so its feature effects are causal for little-coder with all features operational on this benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.