- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In an observational analysis of the Terminal-Bench 2.0 leaderboard, the authors use an additive generalized linear model (GLM) to estimate harness effects while adjusting for LLM identity, and LLM effects while adjusting for harness identity. The estimated gap between the highest- and lowest-ranked harnesses is similar to the gap between Claude Opus 4.1 and Claude Opus 4.6 on the log-odds scale. To test whether harness effects vary across LLMs, the authors analyze nine LLMs evaluated with Mini-SWE-Agent, OpenHands, and Terminus 2. They find that OpenHands' advantage over Mini-SWE-Agent varies across these LLMs.
To examine individual harness features, the authors study little-coder with three LLMs: Gemma-4-26B-A4B (Gemma), K2-Horizon-MoVA-36B-A4B (K2), and Qwen3.6-35B-A3B (Qwen). For each LLM, they randomly select which features to enable and test the resulting configurations on the same Terminal-Bench 2.0 tasks. Infrastructure (INFRA) is always enabled. The analysis excludes configurations without shell session (SS), which provides the shell tool needed to run the tasks. Both the additive model and KernelSHAP find that quality monitoring contributes positively on average across the three LLMs.
Critical strengths
S1: The comparison with LLM upgrades helps interpret the harness effects. Expressing both on the same log-odds scale gives the reader a useful reference for judging the magnitude of harness differences. The authors appropriately distinguish these observational comparisons from causal estimates. The authors also compare OpenHands and Terminus 2 with Mini-SWE-Agent using nine LLMs evaluated with all three harnesses. On this scale, they reject the hypothesis that the OpenHands–Mini-SWE-Agent difference is constant across LLMs, but do not reject the hypothesis of a constant Terminus 2–Mini-SWE-Agent difference.
In little-coder, the quality monitor detects failures such as empty responses and repeated loops and prompts the agent to retry. Across Gemma, K2, and Qwen on Terminal-Bench 2.0, with INFRA and SS enabled, the authors estimate the largest average contribution for quality monitoring in both the additive feature model and KernelSHAP. KernelSHAP estimates feature contributions without imposing additivity, allowing a feature's contribution to depend on which other features are enabled. It gives wider confidence intervals here, but quality monitoring is the only feature with a significantly positive average contribution under both methods. This provides a check on the additive model using the same observations, rather than an independent replication.
S2: The robustness checks support both the harness comparison and the average benefit of quality monitoring. For the Terminal-Bench 2.0 leaderboard, the authors add task fixed effects to their additive harness–LLM model to account for task difficulty. This increases the coefficients but leaves the harness and LLM rankings essentially unchanged. The estimated ForgeCode–Mini-SWE-Agent difference remains comparable to the estimated Claude Opus 4.6–Claude Sonnet 4.5 difference on the log-odds scale. For their binomial GLMs, the authors compare model-based standard errors with trial-row HC0 sandwich standard errors. They find that the HC standard errors are only slightly different and yield qualitatively identical results: the intervals exclude zero for the same coefficients and contrasts under both methods. Both methods still treat trials as independent.
For the little-coder configurations on Terminal-Bench 2.0 with INFRA and SS enabled, the authors also provide confidence intervals for the KernelSHAP feature contributions under an alternative uncertainty estimate. This calculation counts some noise in the observed configuration scores twice. The alternative calculation still supports a positive average contribution of quality monitoring across Gemma, K2, and Qwen: the interval for that average remains entirely above zero. The wide interval leaves the size of that benefit uncertain.
Critical weaknesses
1. The standard errors treat correlated outcomes as independent. Terminal-Bench 2.0 evaluates coding agents on 89 terminal-based tasks, with 5 trials per task, and Each GLM is fitted to trial rows: one binary outcome Yi per harness–LLM–task–repeat trial. The model-based errors assume that the working GLM is correctly specified, yet The additive model in Equation 2.1 has no task term, so shared task difficulty enters its residuals. The paper states this itself: Outcomes on the same task may share a difficulty that the GLMs omit. The sensitivity check does not address it, because Like the model-based errors, the sandwich treats trials as independent, The two estimators therefore differ only through the entry-level lack of fit, and Independence of trials alone therefore does not make HC0 consistent for re-execution variance under mean misspecification. The refit with task fixed effects changes the coefficients (ForgeCode − Mini-SWE-Agent goes from 0.890 to 1.594), but Both fits use model-based standard errors. So the conclusion that No coefficient or contrast changes its significance call, so every bold entry in Tables 1–4 is unchanged compares two estimators that share the same independence assumption. Standard errors clustered by task, or by entry and task, are needed. The per-category fits are affected most: File operations (5) has five tasks, and ForgeCode's 2.121 ± 0.494 is bold.
2. The leaderboard entries are self-selected. The paper states that The fitted effects are adjusted comparisons from an observational leaderboard, not causal estimates. Its headline findings still say that the harness can matter as much as an LLM upgrade and that a better harness can pay off as much as upgrading to a newer flagship LLM, which read as causal. The paper does not discuss how the 118 entries of The leaderboard snapshot we collected were selected by their submitters. Section 5 describes how ForgeCode's model-specific fixes—schema field ordering, flatter schemas, explicit truncation notices, and enforced reviewer-mode verification—brought both GPT-5.4 and Opus 4.6 to 81.8% on Terminal-Bench 2.0. Part of ForgeCode's coefficient may therefore reflect fixes made for this benchmark, not a harness effect in general.
3. The central comparison sets a range against a single contrast. Our central empirical result comes from comparing the harness effects with familiar LLM upgrades: the harness effects span 1.12 logits, from Gemini CLI (−0.20) to Ante (0.93), comparable to recent progress within a flagship line: 1.01 logits from Claude Opus 4.1 to Claude Opus 4.6. The span is the gap between the two most extreme of 28 estimates, while the Opus contrast is one difference. The intervals are reported without adjustment for multiple comparisons or selection of the largest effects, and the span is exactly such a selection. Its top end is Ante, and Ante has the largest estimated harness effect, but it is observed with only one LLM, so its coefficient may also reflect a pairing-specific interaction. This is not a single case, because many harnesses are evaluated on only one or two LLMs. Shrinkage of harness effects, or a span restricted to harnesses with more than one entry, would make the comparison fair.
4. Harness and LLM levels may hide different configurations. After normalizing duplicate LLM names, this corresponds to 39 LLM levels, but the paper does not say whether entries with different reasoning settings or model versions were merged into one level. Leaderboard harness versions are not reported either, although the paper itself notes for little-coder that the two Terminal-Bench entries also differ in harness version. Part of a harness effect may then be an effect of model settings or of harness version.
5. The interaction finding rests on one harness. The abstract states that the harness effects can be heterogeneous; i.e., some harnesses work better with some LLMs than with others. In the 3 × 9 block, The departures come from OpenHands (G² = 63.7, 8 df, p < 10⁻¹⁰) rather than Terminus 2 (G² = 5.5, 8 df, p = 0.71), and for Terminus 2 all of its non-reference intervals include zero. The evidence for heterogeneity therefore comes from one harness. The interactions are negative for Claude Haiku 4.5 and Gemini 2.5 Pro (−0.95 and −0.58), and the paper offers only a possibility: differences in how LLMs handle tool calls interact with OpenHands' action interface, though the success-rate analysis does not identify this mechanism. Error and timeout rates per cell would show whether these values reflect harness–LLM fit or failed runs.
6. The interaction estimates depend on the scale and on the reference. Under our reference coding, the interaction is a difference-in-differences in log-odds, computed only with the logit link, and the paper does not check whether the OpenHands pattern holds on another scale. The paper states that The references are only a coding convention: changing them shifts every harness (or LLM) coefficient by the same constant, leaving the rankings and the differences between coefficients unchanged. This holds for the additive model. In Table 3, however, each γij measures how the harness contrast changes relative to the reference LLM, rather than a residual from the separately fitted additive model, and interaction terms involving either reference level are zero by construction. The reported pattern, positive Relative to Claude Sonnet 4.5 for the GPT-5 models, therefore depends on choosing Claude Sonnet 4.5. Only the likelihood-ratio test does not depend on which harness and LLM serve as references.
7. The KernelSHAP analysis adds complexity without new conclusions. The constraint enforces Shapley efficiency: the nine feature attributions sum to the empirical log-odds difference between the full and baseline configurations, and each configuration is evaluated with one recorded attempt per task and a binary success outcome. For Qwen, the baseline solves 26/89 (29.2%) tasks and the full configuration 21/89 (23.6%), so its nine attributions must sum to −0.290. The paper itself says that KernelSHAP allows feature interactions at the cost of wider intervals. Most of that width is noise in the coalition values, amplified by the endpoint constraint and the concentrated kernel weights. Its variance construction is our approximation, its degrees of freedom are calibration approximations, and clipping guarantees neither an upper bound on the true variance nor nominal finite-sample coverage. The paper concludes that The additive GLM gives the more precise summary, and that the sign changes of EV, KS and TB remain inconclusive. Since the feature GLM excludes interactions among harness features, adding pairwise interactions to it and testing them with a likelihood-ratio test would check additivity more directly.
8. The method itself is standard. The abstract says We develop a simple statistical method to disentangle the effects of the LLM and its harness on the agent's score. The paper also says We use standard generalized linear models (McCullagh & Nelder, 1989) to decompose variation in agent scores into harness and LLM effects and We fit Equation 2.1 to data with off-the-shelf logistic regression software. The Shapley reading adds no new quantity: In our two-attribute reference game these first-order components are exactly the Shapley contributions of harness and LLM identity on the link scale, and Hence, in the additive model, the harness and LLM coefficients are simultaneously the first-order functional components and the ordinary BShap attributions relative to the reference pair. The contribution lies in the application to Terminal-Bench 2.0 and in the collected data. The paper should either present itself as an empirical study or show what the approach gives beyond a standard GLM for a design that is sparse and unbalanced.
Questions for authors
-
Outcomes on the same task may share a difficulty that the GLMs omit. Do the bold entries in Tables 1–4 remain bold with standard errors clustered by task, or by entry and task? How many of the bold coefficients in Table 2 remain? -
The fitted effects are adjusted comparisons from an observational leaderboard. Do you know how the entries inThe leaderboard snapshot we collectedwere produced, in particular whether submitters evaluated several pairings and reported only some of them? -
Ante has the largest estimated harness effect, but it is observed with only one LLM. What is the span of harness effects overharnesses with more than one entryonly? Which LLMs do ForgeCode's two entries use? -
After normalizing duplicate LLM names, this corresponds to 39 LLM levels. Were entries with different reasoning settings or model versions merged into one LLM level? Which harness version does each leaderboard entry use? -
The OpenHands interactions are
negative for Claude Haiku 4.5 and Gemini 2.5 Pro (−0.95 and −0.58). What are the error and timeout rates in these cells, and does the interaction remain after runs that failed for infrastructure reasons are excluded? -
The departures come from OpenHands. Does this remain under another link function, for example probit or identity? How does Table 3 change with another reference LLM, sinceinteraction terms involving either reference level are zero by construction? -
KernelSHAP is applied
To estimate Shapley attributions without imposing additivity. What does a likelihood-ratio test of pairwise feature interactions in the feature GLM show, and what does KernelSHAP add ifThe additive GLM gives the more precise summary? -
The paper uses
standard generalized linear modelsfittedwith off-the-shelf logistic regression software. Which part of thesimple statistical methoddo you consider new, beyond its application to Terminal-Bench 2.0?
3 comments
Response to Reviewer XpAU (1/2)
We thank Reviewer XpAU for the questions and requests for clarification.
W1/Q1: Clustered SEs. Each trial is executed independently, so evaluating multiple agents on the same task does not itself imply dependence between trial outcomes. Moreover, Appendix C already accounts for shared task difficulty through task fixed effects. The resulting harness and LLM rankings remain essentially unchanged (Spearman 0.999/1.000), as do all significance conclusions. HC0 sandwich SEs also yield nearly identical results (median SE ratio 0.999–1.001). As an additional robustness check, we computed task-clustered SEs. All previously significant entries in Tables 1, 3, and 4 remain significant. Clustering by entry is not needed beyond this: all trials of an entry share the same harness–LLM design row, so beyond shared task difficulty they can be correlated only through entry-level lack of fit of the mean model. The 3 × 9 interaction block is saturated and has none, and for the additive model the HC0 SEs, which grow with entry-level lack of fit, are nearly identical to the model-based SEs. For Table 2, most categories contain only 5–9 tasks, too few clusters for reliable cluster-robust inference.
W2/Q2: Entry selection. The leaderboard records only submitted entries, so we cannot determine whether developers evaluated additional pairings that were not reported. We evaluate the submitted harness configurations, including their prompts and tuning, and use the GLM to adjust comparisons for LLM identity. These remain observational comparisons among submitted entries; submission selection may affect the fitted differences. Section 4's controlled experiment is free of this selection: we fixed the three LLMs in advance, randomly sampled the little-coder configurations, and ran and report every configuration on every task.
W3/Q3: Harness span and sparse pairings. Our comparison with an LLM upgrade is intended to provide an intuitive reference for interpreting the magnitude of the estimated harness span, rather than to test the equality of the two contrasts. Both are measured on the same log-odds scale. Restricting to the 17 harnesses evaluated with more than one LLM yields a span of 1.09 logits (Gemini CLI to ForgeCode), still comparable to the Claude Opus 4.1 → 4.6 upgrade (1.01 logits). ForgeCode's two entries use Claude Opus 4.6 and Gemini 3.1 Pro. Thus, the harness span remains comparable to the LLM upgrade even after excluding harnesses evaluated with only one LLM.
W4/Q4: LLM and harness configurations. The leaderboard specifies a common evaluation protocol requiring repeated trials under the same configuration. We did not merge distinct LLM versions; only three naming variants were standardized. Reasoning settings are not separately recorded in the leaderboard, and harness versions are available for only some entries. All OpenHands entries in the 3 × 9 interaction analysis use version 0.60.0.
W5/Q5: Error and timeout rates. The leaderboard reports each trial's outcome only as pass or fail, without distinguishing infrastructure failures, timeouts, or unsuccessful task attempts. Therefore, these rates cannot be separately computed, and infrastructure-related failures cannot be identified and excluded from the analysis.
W6/Q6: Link function and reference coding. The OpenHands interaction remains significant under logit, probit, and identity links (LRT G² = 63.7/63.7/63.6, p < 10⁻¹⁰), while Terminus 2 remains non-significant (G² = 5.5–5.6, p ≥ 0.69). On the probability scale, OpenHands' relative advantage is 11.6 ± 2.1 percentage points larger for GPT-5 models. We agree that individual interaction coefficients depend on the reference LLM, but the differences in harness advantages across LLMs and the overall interaction tests are reference-invariant. Thus, our main conclusions do not depend on the choice of link function or reference coding.
W7/Q7: Feature interactions and KernelSHAP. Adding all 45 pairwise feature interactions to the GLM yields no significant improvement (joint LRT G² = 46.3 on 135 df, p > 0.99). KernelSHAP adds a check that does not impose additivity: it targets Shapley values, which remain interpretable under feature interactions. It agrees with the GLM in sign for quality monitoring and write protection, but its intervals are wider because the Shapley kernel concentrates leverage on a few configurations; these configurations are not outliers. Dropping the endpoint constraints gives the same picture (quality monitoring: 0.077 [−0.029, 0.182] constrained, 0.084 [−0.004, 0.171] unconstrained), so the conclusion does not depend on the constraint; we keep it because it enforces Shapley efficiency. We therefore treat the GLM as the primary analysis and KernelSHAP as a check on it. Please see Appendix D.4 of the revised paper for the full comparison of the two analyses.
Response to Reviewer XpAU (2/2)
W8/Q8: Contribution. We respectfully disagree that the use of established statistical methods limits the novelty or informativeness of our contribution. Our central contributions are important new empirical findings about coding harnesses, not new statistical estimators. While practitioners have long recognized that harnesses matter, the magnitude of their effects, their dependence on the underlying LLM, and the contributions of individual harness features have not been systematically quantified.
Our study establishes three concrete findings:
- Harness choice can be as consequential as an LLM upgrade: the range of harness effects is comparable to the Claude Opus 4.1 → Opus 4.6 upgrade (1.12 vs. 1.01 logits).
- Harness effectiveness depends on the underlying LLM: OpenHands' relative advantage is 0.70 ± 0.13 logits larger for GPT-5 models than for Claude and Gemini models.
- Most harness features have no detectable effect: of ten feature groups, only quality monitoring improves little-coder's average success on Terminal-Bench (0.133 ± 0.030), and write protection even reduces it (−0.066 ± 0.030).
These findings have direct practical implications: harness choice deserves as much attention as LLM choice, harnesses should be tailored to the LLM, and harness features should be measured on the target LLMs and tasks rather than added by default. We revised the third finding in the abstract and introduction to reflect the updated experiment.
Beyond these empirical findings, we carefully quantify statistical uncertainty, examine robustness, and establish a connection between our GLM decomposition and Shapley attribution. We also explored several Shapley estimation approaches and selected the standard formulation as the most suitable for our harness evaluation setting.
If the reviewer knows of prior work that has already addressed these empirical questions (e.g., studies quantifying how much a coding harness contributes to agent performance relative to the LLM), we would appreciate relevant references.
In response to the Program Chairs’ concern about AI use, I used Grammarly to improve the readability of my review. I identified the paper’s strengths and weaknesses myself. I did not use any AI tools to identify those strengths or weaknesses, form my overall assessment, or determine my score.