- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper investigates how semantically equivalent changes to LLM agent harnesses can affect model performance and leaderboard rankings. The authors evaluate 24 models across 32 harness configurations on three multi-turn tasks: text-to-SQL, mathematics, and code generation. They vary aspects of the harness, including prompt wording, reasoning format, action protocols, and instruction ordering, while keeping the underlying tasks and evaluation budgets fixed. Their results show that although harness changes have a relatively small effect on average performance, they can cause substantial accuracy differences for individual models and frequently reverse rankings among closely performing models. To better understand these differences, the authors decompose performance changes into three sources: Delivery, Format, and Content failures, finding that submission and parsing issues account for a substantial portion of the observed degradation. They also validate the phenomenon on an additional tool-use benchmark and introduce ScaffoldSpread, an eight-configuration audit designed to estimate model sensitivity without evaluating the full configuration grid. Overall, the paper argues that leaderboard rankings depend not only on model capabilities but also on harness design, and recommends reporting harness sensitivity alongside benchmark scores.
Critical strengths
-
The paper evaluates a relatively large and diverse set of 24 models across 32 harness configurations and three tasks, with additional validation on BFCL. This provides substantial empirical evidence that harness variations can affect model performance and rankings, particularly among closely performing models.
-
Beyond reporting ranking changes, the authors investigate their sources through variance decomposition, parser interventions, and failure analysis. These analyses provide useful insights into how submission, formatting, and content failures contribute to performance differences, making the findings more informative and practically relevant.
Critical weaknesses
-
My main concern is the novelty of the paper's findings. Prior work has already shown that small changes in prompts and evaluation formats can significantly affect model performance and rankings, and related studies have also investigated the influence of agent scaffolds on evaluation outcomes. Although this paper extends these observations to multi-turn harness configurations, I am not fully convinced that it provides sufficiently new insights beyond existing work. The paper would benefit from clearer differentiation from prior studies, particularly those focusing on agent scaffolds and evaluation sensitivity.
-
Another major concern is whether the observed instability reflects genuine harness sensitivity or limitations of the evaluation pipeline. The authors show that a substantial portion of the performance degradation comes from parsing and submission failures rather than incorrect solutions. Although they provide detailed failure analyses, it remains unclear how much of the reported leaderboard instability would persist with more robust parsing and submission handling. Furthermore, some supposedly equivalent harness changes, such as switching between native tool calls and text-based protocols or removing turn reminders, alter the interaction interface or information available to the model rather than just presentation. These issues raise questions about the validity of the experimental setup and how broadly its conclusions can be interpreted.
Questions for authors
-
How much of the reported leaderboard instability would remain with more robust parsing and standardized submission handling? Although the paper provides detailed failure analyses, a substantial portion of the performance degradation comes from parsing and submission failures rather than incorrect solutions. Could the authors clarify whether the main ranking results would still hold after accounting for these issues, and to what extent the observed instability reflects harness sensitivity rather than limitations of the evaluation pipeline?
-
Could the authors clarify how the main findings differ from prior work on prompt-format sensitivity, leaderboard instability, and the impact of agent scaffolds on evaluation? In particular, what new insights does the multi-turn harness setting provide beyond previously established observations? How does the proposed ScaffoldSpread method differ from existing approaches to measuring evaluation sensitivity?
-
How do the authors justify treating all seven harness dimensions as semantically equivalent? Some changes, such as switching between native tool calling and text-based protocols or removing turn reminders, appear to modify the interaction interface or information available to the model rather than just presentation. Would the main findings still hold when considering only strictly presentation-level changes?
-
Given that the main experiments use relatively short interactions of at most five action–observation turns, how confident are the authors that the findings generalize to more realistic agent settings involving longer trajectories, complex tool use, and recovery mechanisms? Does the additional BFCL experiment provide enough evidence to support the broader claims about agent benchmarks?
-
Given that random coverage-constrained subsets already achieve relatively high correlations with full-grid sensitivity rankings, what is the practical advantage of the proposed selection procedure over simpler sampling strategies? Also, how well would the selected configurations generalize to unseen models or independently constructed harness variants, considering that the current cross-domain experiments reuse the same model pool?
-
The paper reports that most adjacent model pairs can be reversed under at least one harness configuration. However, how often do these reversals occur across configurations, and how many involve statistically meaningful performance differences rather than nearly tied models? Could the authors clarify how these factors affect the practical significance of the reported leaderboard instability?
4 comments
Response to Reviewer Epy4 (2/3)
W2 (semantic equivalence) / Q3: Restricting changes to presentation
Would the main findings still hold when considering only strictly presentation-level changes?
Our equivalence criterion preserves task information, available tool capabilities, and the execution budget. Native and text-based calls expose the same tools and operations through different encodings. Turn reminders restate the remaining budget, which is already determined by the announced limit and the interaction history. Thus, these changes alter how existing capabilities and information are presented to the model while keeping the task to be solved the same.
We also directly tested the reviewer’s proposed restriction by excluding native-function-call configurations and keeping the turn reminder on throughout. The resulting 18-configuration subgrid fixes both disputed choices and varies the remaining presentation dimensions.
On code, mean Kendall ranking agreement is 0.685, almost identical to 0.686 in the full grid, and GPT-5.6 still spans 21 ranking positions. Mean agreement is 0.870 on math and 0.822 on SQL. The same ranking instability therefore appears under the restricted comparison, reinforcing the conclusion that presentation choices affect model comparisons.
Q4: Multi-turn agent and terminal workflows
Does the additional BFCL experiment provide enough evidence to support the broader claims about agent benchmarks?
We now have evidence from both multi-turn tool-use and terminal tasks. The paper’s BFCL v3 experiment uses 50 multi-turn tasks and six API models under 20 harness configurations, with the official state checker; the score range is 24–40 percentage points per model (Table 2). The additional Terminal-Bench experiment allows up to 12 action–feedback rounds in shell-based workflows, where models issue commands, inspect execution feedback, and revise their actions.
We additionally evaluated four models on the same 57 Terminal-Bench 1.0 tasks with four harness variants (912 task–model–variant cells). V1 uses baseline instructions with shell commands in Markdown code blocks and a standalone DONE completion marker. V2 paraphrases the instructions while keeping that protocol. V3 uses lowercase <think>, <cmd>, and <done/> tags; V4 changes only those tags to uppercase <THINK>, <CMD>, and <DONE/>.
| Model | V1 | V2 | V3 | V4 | Std. (pp) | Range (pp) |
|---|---|---|---|---|---|---|
| GPT-5.6 | 73.7% | 63.2% | 66.7% | 50.9% | 9.5 | 22.8 |
| Claude-Sonnet | 36.8% | 49.1% | 15.8% | 12.3% | 17.5 | 36.8 |
| Claude-Opus | 66.7% | 70.2% | 61.4% | 63.2% | 3.9 | 8.8 |
| GLM-5.3 | 29.8% | 38.6% | 33.3% | 29.8% | 4.1 | 8.8 |
Each entry is accuracy on the common 57-task set. Sample standard deviation and range summarize the four variant accuracies.
GPT-5.6 falls from 73.7% under V1 to 50.9% under V4, a 22.8-point difference. Thirteen tasks change from pass to fail and none change in the reverse direction. The effect appears in both task batches: 6:0 in the first and 7:0 in the second. GPT-5.6 also moves from above Opus under V1 to below it under V4. These results show that harness choice affects both success rates and model comparisons in terminal workflows.
BFCL and Terminal-Bench therefore extend the controlled short-task findings to agent interaction with external tools and executable environments. This directly supports the relevance of harness sensitivity to agent evaluation.
Response to Reviewer Epy4 (3/3)
Q5: Selection versus random sampling and transfer
Given that random coverage-constrained subsets already achieve relatively high correlations with full-grid sensitivity rankings, what is the practical advantage of the proposed selection procedure over simpler sampling strategies?
At the same eight-configuration budget, selection improves the recovery of sensitivity rankings over random coverage. The fixed audit achieves ρ = 0.9812 on the evaluated grid, compared with a median of 0.9391 across all 299,473 subsets satisfying the same anchor and dimension-coverage constraints. It exceeds more than 99.99% of those subsets. The practical gain is more accurate identification of harness-sensitive models for the same number of model evaluations.
The held-out-domain comparison shows the same advantage (Table 8). Selecting on two domains and testing on the third gives 0.9609 versus the random-subset median of 0.9157 on SQL, 0.9678 versus 0.9496 on math, and 0.9191 versus 0.8948 on code. Thus, selecting configurations improves on the typical coverage-constrained random subset in all three domain transfers.
Table 8 measures held-out-domain transfer within the same 24-model pool. For a new model, the fixed eight configurations provide a sensitivity estimate at the same reduced cost; its accuracy can be assessed against the full grid on a held-out model cohort. Independently constructed harness variants define a new candidate space, to which the same coverage-constrained selection procedure can be applied. The reusable contribution is therefore both a fixed audit for the studied harness space and a procedure for selecting audits in other spaces.
Q6: Frequency, size, and practical significance of reversals
However, how often do these reversals occur across configurations, and how many involve statistically meaningful performance differences rather than nearly tied models?
The reversals occur repeatedly and have substantial score gaps. Ordering code models by their median score across configurations and excluding adjacent pairs with equal medians leaves 20 adjacent pairs. Eighteen reverse under at least one configuration. Across the 20 pairs, the lower-median model wins under a median of 11.5 of the 32 configurations (35.9%), with counts ranging from 0 to 16.
All 18 reversing adjacent pairs have at least one reversal with a score gap above 3.5 percentage points. For scale, the paper’s five-repeat experiment holds the tasks and configuration fixed and finds a median within-cell standard deviation of 1.4 points and a pooled RMS of 2.0 points; day-apart retests of the API models shift scores by at most two points (Section 3.2). The 3.5-point filter selects reversals with score differences large relative to the typical repeat-run variation.
The full code grid provides a broader check: 157 of 276 model pairs strictly reverse, and 138 of those 157 show score advantages exceeding 3.5 points in both directions under different configurations. Among the reversing pairs, the median fraction of configuration pairs producing opposite model orderings is 26.2%. The recurring changes in order, together with their score gaps, motivate reporting harness sensitivity alongside a leaderboard’s point scores. This directly affects model selection: a model chosen for its lead under one harness can fall behind under another equivalent harness. Reporting variation across harnesses helps users judge how much a reported advantage depends on the evaluation interface.
Thank you for the detailed responses and additional experiments. I appreciate the effort to address my concerns, and I think most of them have been substantially addressed, particularly regarding semantic equivalence, generalizability, and the significance of ranking instability.
However, I still have two main concerns. First, I am not fully convinced that the paper provides sufficiently new insights beyond existing work on prompt sensitivity and agent scaffolds. The response clarifies the intended contributions but does not clearly establish how they differ from prior findings. Second, although the additional experiments show that harness sensitivity persists with more robust evaluation, it remains unclear how much of the original leaderboard instability is attributable to avoidable parsing and submission issues rather than harness sensitivity itself.
Overall, while the new evidence strengthens the paper and addresses several of my concerns, I don't think it fully resolves these two issues, which remain important to my assessment. Therefore, I have decided to keep my original score.
Response to Reviewer Epy4 (1/3)
Thank you for the detailed questions about the contribution, evaluation controls, and practical interpretation of the results.
W2 (parsing and submission) / Q1: Rankings after interface repair
Model-order reversals persist after applying robust extraction and standardizing the tool interface. We directly tested this with native tool calling and L1, a tolerant parser that removes outer code fences and tags. On the same 164 code tasks for 23 models, two configurations hold the interface, prompt structure, wording, and turn reminders fixed and vary only reasoning format, tool-name capitalization, and rule order. Eleven of 253 model pairs reverse their score ordering; six models change accuracy by more than 10 points, and the largest average-rank change is 5.5 places.
The failure analysis explains the mechanisms behind this dependence. Among the 154 code model–configuration cells more than seven points below the anchor, Delivery, Format, and Content account for 45%, 40%, and 15% of the signed score loss (Table 7). These correspond to unsuccessful submission, extraction loss, and incorrect submitted content. Re-scoring the same saved code outputs with tolerant extraction raises the pass rate from 78.6% to 86.3%, removing 36.2% of strict-parser failures. Twenty-one of the 32 configurations gain at most 10 points; only three gain more than 20.
Together, the repaired-interface experiment and the fixed-output replay establish the paper’s central result: measured model comparisons depend on the harness even after the identified extraction issue is addressed. The decomposition makes that dependence actionable by showing which stage an evaluator should inspect.
W1 / Q2: New insights and the role of ScaffoldSpread
Our multi-turn setting connects harness changes to the process that produces the final score. In an open-ended tool-use task, a model must construct actions, use feedback, and deliver a solution through a protocol. A change in that protocol can shift failures between unsuccessful submission, extraction, and incorrect content. Our Delivery/Format/Content decomposition tracks those shifts.
The analysis also identifies which model comparisons are most exposed. In the code experiments, score consistency is high across the full model pool (ICC 0.79) but low among frontier models (0.29). This explains how a leaderboard can preserve the broad separation between strong and weak models while failing to reliably order close competitors. The relevant quantity is the size of model differences relative to harness variation, which tracks consistency across groups and domains with ρ = 0.93.
ScaffoldSpread addresses the cost of measuring this dependence. FormatSpread (Sclar et al., 2024) estimates the performance range of a model over prompt formats; our target is the ranking of models by their sensitivity, measured as the standard deviation of accuracy across harness configurations. An eight-configuration audit recovers this ranking with ρ = 0.9812, using one quarter of the full-grid configuration budget. This enables leaderboard publishers to report performance together with its variability across harnesses, much as a mean is reported alongside a standard deviation. Readers can then distinguish consistently strong performance from an advantage tied to one harness.
Relative to agent-scaffold comparisons, our controlled design fixes task goals, available tool capabilities, and budgets while varying the protocol used to access them. It therefore isolates the dependence of model comparisons on equivalent harness choices. BFCL v3 and the additional Terminal-Bench results connect that controlled finding to multi-turn agent benchmarks.