acceptodds
Under review as a conference paper at ICLR 2027

The Emperor's New Scaffold: Semantically Equivalent Harnesses Reshuffle the Leaderboard

Abstract

Every new language model arrives with a leaderboard that crowns it, and the community reads these tables as measurements of the models themselves. But does a leaderboard for agentic tasks measure the model, or the model together with the harness of prompts and protocol it was run under? We show that a one-word, semantically equivalent edit to the harness drops a flagship model's accuracy from 93.3 to 14.0 on code generation, below all five sub-2B models we tested. To test whether this is an isolated failure, we evaluate 24 models under 32 equivalent harness variants on three multi-turn agentic tasks (text-to-SQL, math, code). Nearly every pair of closely ranked models can be flipped by some equivalent rewrite, with swings far beyond resampling noise, and equivalent harnesses disagree on the top-3 in 47% to 84% of configurations across the three tasks. This instability lands exactly where it costs the most. Nobody consults a leaderboard to learn that a frontier model beats a sub-1B one; they consult it to choose between close competitors. Tracing every deviation through an exact decomposition, we find three failure modes, Delivery, Format, and Content; which one fails depends on which model meets which rewrite, and no harness in our grid treats all models impartially. Harness rewrites cannot move mountains, but they reshuffle the close races. We therefore release ScaffoldSpread, an eight-configuration audit that reproduces full-grid sensitivity at ρ=0.981, together with the parameterized harness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent evaluation, scaffold sensitivity, multi-turn agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Epy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how semantically equivalent changes to LLM agent harnesses can affect model performance and leaderboard rankings. The authors evaluate 24 models across 32 harness configurations on three multi-turn tasks: text-to-SQL, mathematics, and code generation. They vary aspects of the harness, including prompt wording, reasoning format, action protocols, and instruction ordering, while keeping the underlying tasks and evaluation budgets fixed. Their results show that although harness changes have a relatively small effect on average performance, they can cause substantial accuracy differences for individual models and frequently reverse rankings among closely performing models. To better understand these differences, the authors decompose performance changes into three sources: Delivery, Format, and Content failures, finding that submission and parsing issues account for a substantial portion of the observed degradation. They also validate the phenomenon on an additional tool-use benchmark and introduce ScaffoldSpread, an eight-configuration audit designed to estimate model sensitivity without evaluating the full configuration grid. Overall, the paper argues that leaderboard rankings depend not only on model capabilities but also on harness design, and recommends reporting harness sensitivity alongside benchmark scores.

Critical strengths
  • The paper evaluates a relatively large and diverse set of 24 models across 32 harness configurations and three tasks, with additional validation on BFCL. This provides substantial empirical evidence that harness variations can affect model performance and rankings, particularly among closely performing models.

  • Beyond reporting ranking changes, the authors investigate their sources through variance decomposition, parser interventions, and failure analysis. These analyses provide useful insights into how submission, formatting, and content failures contribute to performance differences, making the findings more informative and practically relevant.

Critical weaknesses
  • My main concern is the novelty of the paper's findings. Prior work has already shown that small changes in prompts and evaluation formats can significantly affect model performance and rankings, and related studies have also investigated the influence of agent scaffolds on evaluation outcomes. Although this paper extends these observations to multi-turn harness configurations, I am not fully convinced that it provides sufficiently new insights beyond existing work. The paper would benefit from clearer differentiation from prior studies, particularly those focusing on agent scaffolds and evaluation sensitivity.

  • Another major concern is whether the observed instability reflects genuine harness sensitivity or limitations of the evaluation pipeline. The authors show that a substantial portion of the performance degradation comes from parsing and submission failures rather than incorrect solutions. Although they provide detailed failure analyses, it remains unclear how much of the reported leaderboard instability would persist with more robust parsing and submission handling. Furthermore, some supposedly equivalent harness changes, such as switching between native tool calls and text-based protocols or removing turn reminders, alter the interaction interface or information available to the model rather than just presentation. These issues raise questions about the validity of the experimental setup and how broadly its conclusions can be interpreted.

Questions for authors
  • How much of the reported leaderboard instability would remain with more robust parsing and standardized submission handling? Although the paper provides detailed failure analyses, a substantial portion of the performance degradation comes from parsing and submission failures rather than incorrect solutions. Could the authors clarify whether the main ranking results would still hold after accounting for these issues, and to what extent the observed instability reflects harness sensitivity rather than limitations of the evaluation pipeline?

  • Could the authors clarify how the main findings differ from prior work on prompt-format sensitivity, leaderboard instability, and the impact of agent scaffolds on evaluation? In particular, what new insights does the multi-turn harness setting provide beyond previously established observations? How does the proposed ScaffoldSpread method differ from existing approaches to measuring evaluation sensitivity?

  • How do the authors justify treating all seven harness dimensions as semantically equivalent? Some changes, such as switching between native tool calling and text-based protocols or removing turn reminders, appear to modify the interaction interface or information available to the model rather than just presentation. Would the main findings still hold when considering only strictly presentation-level changes?

  • Given that the main experiments use relatively short interactions of at most five action–observation turns, how confident are the authors that the findings generalize to more realistic agent settings involving longer trajectories, complex tool use, and recovery mechanisms? Does the additional BFCL experiment provide enough evidence to support the broader claims about agent benchmarks?

  • Given that random coverage-constrained subsets already achieve relatively high correlations with full-grid sensitivity rankings, what is the practical advantage of the proposed selection procedure over simpler sampling strategies? Also, how well would the selected configurations generalize to unseen models or independently constructed harness variants, considering that the current cross-domain experiments reuse the same model pool?

  • The paper reports that most adjacent model pairs can be reversed under at least one harness configuration. However, how often do these reversals occur across configurations, and how many involve statistically meaningful performance differences rather than nearly tied models? Could the authors clarify how these factors affect the practical significance of the reported leaderboard instability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Epy4 (1/3)

Thank you for the detailed questions about the contribution, evaluation controls, and practical interpretation of the results.

W2 (parsing and submission) / Q1: Rankings after interface repair

How much of the reported leaderboard instability would remain with more robust parsing and standardized submission handling?

Model-order reversals persist after applying robust extraction and standardizing the tool interface. We directly tested this with native tool calling and L1, a tolerant parser that removes outer code fences and tags. On the same 164 code tasks for 23 models, two configurations hold the interface, prompt structure, wording, and turn reminders fixed and vary only reasoning format, tool-name capitalization, and rule order. Eleven of 253 model pairs reverse their score ordering; six models change accuracy by more than 10 points, and the largest average-rank change is 5.5 places.

The failure analysis explains the mechanisms behind this dependence. Among the 154 code model–configuration cells more than seven points below the anchor, Delivery, Format, and Content account for 45%, 40%, and 15% of the signed score loss (Table 7). These correspond to unsuccessful submission, extraction loss, and incorrect submitted content. Re-scoring the same saved code outputs with tolerant extraction raises the pass rate from 78.6% to 86.3%, removing 36.2% of strict-parser failures. Twenty-one of the 32 configurations gain at most 10 points; only three gain more than 20.

Together, the repaired-interface experiment and the fixed-output replay establish the paper’s central result: measured model comparisons depend on the harness even after the identified extraction issue is addressed. The decomposition makes that dependence actionable by showing which stage an evaluator should inspect.

W1 / Q2: New insights and the role of ScaffoldSpread

In particular, what new insights does the multi-turn harness setting provide beyond previously established observations? How does the proposed ScaffoldSpread method differ from existing approaches to measuring evaluation sensitivity?

Our multi-turn setting connects harness changes to the process that produces the final score. In an open-ended tool-use task, a model must construct actions, use feedback, and deliver a solution through a protocol. A change in that protocol can shift failures between unsuccessful submission, extraction, and incorrect content. Our Delivery/Format/Content decomposition tracks those shifts.

The analysis also identifies which model comparisons are most exposed. In the code experiments, score consistency is high across the full model pool (ICC 0.79) but low among frontier models (0.29). This explains how a leaderboard can preserve the broad separation between strong and weak models while failing to reliably order close competitors. The relevant quantity is the size of model differences relative to harness variation, which tracks consistency across groups and domains with ρ = 0.93.

ScaffoldSpread addresses the cost of measuring this dependence. FormatSpread (Sclar et al., 2024) estimates the performance range of a model over prompt formats; our target is the ranking of models by their sensitivity, measured as the standard deviation of accuracy across harness configurations. An eight-configuration audit recovers this ranking with ρ = 0.9812, using one quarter of the full-grid configuration budget. This enables leaderboard publishers to report performance together with its variability across harnesses, much as a mean is reported alongside a standard deviation. Readers can then distinguish consistently strong performance from an advantage tied to one harness.

Relative to agent-scaffold comparisons, our controlled design fixes task goals, available tool capabilities, and budgets while varying the protocol used to access them. It therefore isolates the dependence of model comparisons on equivalent harness choices. BFCL v3 and the additional Terminal-Bench results connect that controlled finding to multi-turn agent benchmarks.

AuthorsOpenReview ↗

Response to Reviewer Epy4 (2/3)

W2 (semantic equivalence) / Q3: Restricting changes to presentation

Would the main findings still hold when considering only strictly presentation-level changes?

Our equivalence criterion preserves task information, available tool capabilities, and the execution budget. Native and text-based calls expose the same tools and operations through different encodings. Turn reminders restate the remaining budget, which is already determined by the announced limit and the interaction history. Thus, these changes alter how existing capabilities and information are presented to the model while keeping the task to be solved the same.

We also directly tested the reviewer’s proposed restriction by excluding native-function-call configurations and keeping the turn reminder on throughout. The resulting 18-configuration subgrid fixes both disputed choices and varies the remaining presentation dimensions.

On code, mean Kendall ranking agreement is 0.685, almost identical to 0.686 in the full grid, and GPT-5.6 still spans 21 ranking positions. Mean agreement is 0.870 on math and 0.822 on SQL. The same ranking instability therefore appears under the restricted comparison, reinforcing the conclusion that presentation choices affect model comparisons.

Q4: Multi-turn agent and terminal workflows

Does the additional BFCL experiment provide enough evidence to support the broader claims about agent benchmarks?

We now have evidence from both multi-turn tool-use and terminal tasks. The paper’s BFCL v3 experiment uses 50 multi-turn tasks and six API models under 20 harness configurations, with the official state checker; the score range is 24–40 percentage points per model (Table 2). The additional Terminal-Bench experiment allows up to 12 action–feedback rounds in shell-based workflows, where models issue commands, inspect execution feedback, and revise their actions.

We additionally evaluated four models on the same 57 Terminal-Bench 1.0 tasks with four harness variants (912 task–model–variant cells). V1 uses baseline instructions with shell commands in Markdown code blocks and a standalone DONE completion marker. V2 paraphrases the instructions while keeping that protocol. V3 uses lowercase <think>, <cmd>, and <done/> tags; V4 changes only those tags to uppercase <THINK>, <CMD>, and <DONE/>.

ModelV1V2V3V4Std. (pp)Range (pp)
GPT-5.673.7%63.2%66.7%50.9%9.522.8
Claude-Sonnet36.8%49.1%15.8%12.3%17.536.8
Claude-Opus66.7%70.2%61.4%63.2%3.98.8
GLM-5.329.8%38.6%33.3%29.8%4.18.8

Each entry is accuracy on the common 57-task set. Sample standard deviation and range summarize the four variant accuracies.

GPT-5.6 falls from 73.7% under V1 to 50.9% under V4, a 22.8-point difference. Thirteen tasks change from pass to fail and none change in the reverse direction. The effect appears in both task batches: 6:0 in the first and 7:0 in the second. GPT-5.6 also moves from above Opus under V1 to below it under V4. These results show that harness choice affects both success rates and model comparisons in terminal workflows.

BFCL and Terminal-Bench therefore extend the controlled short-task findings to agent interaction with external tools and executable environments. This directly supports the relevance of harness sensitivity to agent evaluation.

AuthorsOpenReview ↗

Response to Reviewer Epy4 (3/3)

Q5: Selection versus random sampling and transfer

Given that random coverage-constrained subsets already achieve relatively high correlations with full-grid sensitivity rankings, what is the practical advantage of the proposed selection procedure over simpler sampling strategies?

At the same eight-configuration budget, selection improves the recovery of sensitivity rankings over random coverage. The fixed audit achieves ρ = 0.9812 on the evaluated grid, compared with a median of 0.9391 across all 299,473 subsets satisfying the same anchor and dimension-coverage constraints. It exceeds more than 99.99% of those subsets. The practical gain is more accurate identification of harness-sensitive models for the same number of model evaluations.

The held-out-domain comparison shows the same advantage (Table 8). Selecting on two domains and testing on the third gives 0.9609 versus the random-subset median of 0.9157 on SQL, 0.9678 versus 0.9496 on math, and 0.9191 versus 0.8948 on code. Thus, selecting configurations improves on the typical coverage-constrained random subset in all three domain transfers.

Table 8 measures held-out-domain transfer within the same 24-model pool. For a new model, the fixed eight configurations provide a sensitivity estimate at the same reduced cost; its accuracy can be assessed against the full grid on a held-out model cohort. Independently constructed harness variants define a new candidate space, to which the same coverage-constrained selection procedure can be applied. The reusable contribution is therefore both a fixed audit for the studied harness space and a procedure for selecting audits in other spaces.

Q6: Frequency, size, and practical significance of reversals

However, how often do these reversals occur across configurations, and how many involve statistically meaningful performance differences rather than nearly tied models?

The reversals occur repeatedly and have substantial score gaps. Ordering code models by their median score across configurations and excluding adjacent pairs with equal medians leaves 20 adjacent pairs. Eighteen reverse under at least one configuration. Across the 20 pairs, the lower-median model wins under a median of 11.5 of the 32 configurations (35.9%), with counts ranging from 0 to 16.

All 18 reversing adjacent pairs have at least one reversal with a score gap above 3.5 percentage points. For scale, the paper’s five-repeat experiment holds the tasks and configuration fixed and finds a median within-cell standard deviation of 1.4 points and a pooled RMS of 2.0 points; day-apart retests of the API models shift scores by at most two points (Section 3.2). The 3.5-point filter selects reversals with score differences large relative to the typical repeat-run variation.

The full code grid provides a broader check: 157 of 276 model pairs strictly reverse, and 138 of those 157 show score advantages exceeding 3.5 points in both directions under different configurations. Among the reversing pairs, the median fraction of configuration pairs producing opposite model orderings is 26.2%. The recurring changes in order, together with their score gaps, motivate reporting harness sensitivity alongside a leaderboard’s point scores. This directly affects model selection: a model chosen for its lead under one harness can fall behind under another equivalent harness. Reporting variation across harnesses helps users judge how much a reported advantage depends on the evaluation interface.

Thank you for the detailed responses and additional experiments. I appreciate the effort to address my concerns, and I think most of them have been substantially addressed, particularly regarding semantic equivalence, generalizability, and the significance of ranking instability.

However, I still have two main concerns. First, I am not fully convinced that the paper provides sufficiently new insights beyond existing work on prompt sensitivity and agent scaffolds. The response clarifies the intended contributions but does not clearly establish how they differ from prior findings. Second, although the additional experiments show that harness sensitivity persists with more robust evaluation, it remains unclear how much of the original leaderboard instability is attributable to avoidable parsing and submission issues rather than harness sensitivity itself.

Overall, while the new evidence strengthens the paper and addresses several of my concerns, I don't think it fully resolves these two issues, which remain important to my assessment. Therefore, I have decided to keep my original score.

Reviewer_LMFLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses the sensitivity of modern benchmarks to the harnesses. It shows that under semantically equivalent modification to the harness used in the evaluation, the flagship models' accuracy may drop significantly and the relative ranking of models in the benchmark can be significantly changed. Further, the paper analyzes three failure modes: Delivery, format and content, and attributes the failure cases into these three categories.

Critical strengths
  1. The paper provides a rigourous empirical evaluation on the sensitivity of agents evaluation benchmark to the harness used, providing insights on the evaluation of latest models. Analysis reveals failure modes of models under semantically equivalent harness modification.
Critical weaknesses
  1. The discussion with more recent related work is somehow missing. In related work section, most works discussed are released in or before 2024, while recently some other works (e.g. [1] ) also propose similar concern that LLM leaderboard results are sensitive the the choice of harnesses. A detailed comparison between more recent works will be needed to illustrate the novelty of the paper.
  2. The significance of the impact of harness sensitivity the leaderboard may be less than expected. From Fig. 2, my intuition is that most of the accuracy drops results from some extreme value, where the configuration causes "bug" in the model. For example, the nested tag problem illustrated in the paper. While this is a worth noting phenomenon in model evaluation, its impact to current leaderboard may be overstated, as it can be well detected, explained and avoided.

[1] There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items, Arxiv 2608.21382.

Questions for authors
  1. Can you provide a detailed comparison of the paper to more recent works, e.g. the one in weakness sections?
  2. How much portion of the observed swings are from a specifc configuration? For example, if we remove one configuration that causes the largest swing, will we still observe similar reranks in the leaderboard beyond the noise?
1 comment
AuthorsOpenReview ↗

Response to Reviewer LMFL

Thank you for highlighting the comparison with recent work and the role of extreme configurations. The results below address both concerns.

W1 / Q1: Comparison with recent related work

Can you provide a detailed comparison of the paper to more recent works, e.g. the one in weakness sections?

The central distinction is the evaluation setting: There Is No Neutral Harness (arXiv:2608.21382) studies correctness changes in fixed-choice questions, whereas we study open-ended generation and multi-turn tool use. The cited paper explicitly scopes out open-ended generation, code, and agentic tasks. In our setting, a harness governs an interaction: the model generates actions, receives tool feedback, and submits an executable solution.

This changes both the question and the analysis. Our analysis follows the interaction through to the reliability of model comparisons: which competitors become difficult to distinguish? Code-domain consistency falls from ICC = 0.79 across the full model pool to 0.29 among frontier models. Multi-turn traces then locate the source of score changes in delivery, extraction, and solution content; fixed-output re-scoring measures the extraction component. Finally, ScaffoldSpread turns this diagnosis into an audit: eight configurations recover the full-grid sensitivity ranking with ρ = 0.981.

The distinction is also supported by agent-benchmark results: BFCL v3 gives 24–40-point score ranges across harnesses (Table 2), and our additional Terminal-Bench study finds a 22.8-point difference for GPT-5.6 between two command-presentation formats. The contribution is therefore a controlled account of how multi-turn evaluation changes model comparisons, supported by failure analysis and a low-cost audit.

W2 / Q2: Sensitivity after removing extreme configurations

For example, if we remove one configuration that causes the largest swing, will we still observe similar reranks in the leaderboard beyond the noise?

The score variation and reranking remain after removing the most extreme configurations. We deleted each of the 32 configurations in turn. Even the deletion that improves code-domain ranking agreement the most changes mean Kendall τ only from 0.686 to 0.697. The mean across models of the cross-configuration accuracy standard deviation changes from 10.34 to 9.99 percentage points.

Removing the three configurations with the largest average score departures from each model’s median gives mean τ = 0.696 and a mean accuracy standard deviation of 9.38 points; GPT-5.6 still spans 21 ranking positions. Thus, substantial variation remains across the other 29 configurations.

We also tested the proposed remedy directly: native tool calling with L1 tolerant extraction, which removes outer code fences and tags. Holding the interface, reminder setting, prompt structure, and wording fixed, changes to reasoning format, tool-name capitalization, and rule order still reverse 11 of 253 model pairs; six of the 23 models change accuracy by more than 10 points. Sensitivity therefore persists after both outlier removal and the proposed formatting repair, making it a property to measure across configurations rather than a single bug to eliminate.

Reviewer_qtm4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies whether agentic leaderboards measure models or model–harness combinations. It evaluates 24 models under 32 semantically equivalent harness configurations on three short multi-turn tasks (text-to-SQL, math with a Python tool, and code generation). Harness effects are small on average but large in specific model–configuration cells, often enough to reverse closely ranked models. The paper decomposes score changes into Delivery, Format, and Content failures, replicates the instability on BFCL, and proposes ScaffoldSpread, an eight-configuration audit that ranks models by harness sensitivity.

Critical strengths

The study covers 24 models, 32 configurations, and three tasks (2,304 cells), and adds an external validation on BFCL with its official checker. Repeated reruns of the most deviant cells, item resampling, and day-apart retests of API models establish a noise floor well below the reported harness effects, which rules out run-to-run variation as an explanation.

Critical weaknesses
  1. Real leaderboards typically evaluate each model through its recommended interface and harness (e.g., native tool calls) with tolerant answer extraction, rather than under arbitrary equivalent rewrites. Moreover, the sensitivity of model performance to prompts and harnesses is largely expected given prior work. Finally, the evaluated tasks (HumanEval, BIRD, and MATH/AIME with at most five turns) are unrelated to the agentic benchmarks the paper uses as motivation (e.g., Terminal-Bench, OSWorld, GDPval), and are not the agentic tasks on which models are widely evaluated today.
  2. The audit only ranks models by their harness sensitivity; it neither estimates accuracy nor recovers leaderboard rankings. Practitioners mainly need the latter, so the audit’s practical value is limited.
Questions for authors

Please see weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qtm4 (1/3)

Thank you for raising the questions about recommended interfaces, agent benchmarks, and the practical role of sensitivity auditing.

W1(a): Recommended interfaces and tolerant extraction

Real leaderboards typically evaluate each model through its recommended interface and harness (e.g., native tool calls) with tolerant answer extraction, rather than under arbitrary equivalent rewrites.

Ranking changes persist with native tool calling and tolerant extraction. We compared 23 models on the same 164 code tasks using L1, a tolerant parser that removes outer code fences and tags. Both configurations use native tool calling, flat prompts, identical wording, and no turn reminders. One uses chain-of-thought formatting, lowercase tool names, and action-first rule order; the other uses tagged reasoning, uppercase tool names, and submission-first rule order.

Across the 253 model pairs, 11 reverse their score ordering. Six models change accuracy by more than 10 percentage points, and the largest average-rank change is 5.5 places. For example, Qwen2.5-7B changes from 53.7% to 37.8%, while Llama-3.2-3B changes from 9.8% to 26.8%. The same presentation change therefore helps some models and hurts others, even with the interface and extraction method fixed.

Even when each model uses its own recommended harness, the leaderboard compares model–harness pairs. The controlled comparison above shows why this matters: presentation choices affect the size and direction of model differences. Reporting harness sensitivity alongside accuracy helps readers judge how much the comparison depends on these choices. Making that dependence measurable and inexpensive to audit is a central contribution of our paper.

AuthorsOpenReview ↗

Response to Reviewer qtm4 (2/3)

W1(b): What the multi-turn analysis contributes

Moreover, the sensitivity of model performance to prompts and harnesses is largely expected given prior work.

Our contribution is to explain when harness sensitivity changes the conclusions drawn from a leaderboard, identify the interaction stages responsible, and make this dependence affordable to audit. Multi-turn tool use gives us both the model comparisons and the execution traces needed to answer these questions.

First, the risk is concentrated in close comparisons. On code, score consistency across configurations, measured by ICC, is 0.79 across all models but only 0.29 within the frontier group. A leaderboard can therefore appear reliable when strong and weak models are pooled, while providing an unreliable ordering among the models users most want to compare. Across the 12 model-group/domain combinations, the ratio of between-model score spread to harness variation tracks consistency with Spearman ρ = 0.93.

Second, a harness changes how a model delivers its solution, how that solution is extracted, and whether the submitted content is correct. A model may generate useful code yet fail to submit it through the required protocol; under another harness, the same model may submit successfully but make a content error. Our Delivery/Format/Content decomposition tracks these changes in failure type, and re-scoring fixed outputs with a tolerant parser isolates the extraction contribution. This connects score variation to specific stages of agent–environment interaction.

Third, ScaffoldSpread makes this diagnosis practical by identifying harness-sensitive models at a smaller evaluation budget. The W2 response below explains its cost and practical value.

W1(c): Evidence from agent benchmarks

Finally, the evaluated tasks (HumanEval, BIRD, and MATH/AIME with at most five turns) are unrelated to the agentic benchmarks the paper uses as motivation (e.g., Terminal-Bench, OSWorld, GDPval), and are not the agentic tasks on which models are widely evaluated today.

The paper already evaluates multi-turn agent tasks on BFCL v3, and our new Terminal-Bench experiment extends this evidence to terminal workflows. On BFCL, six API models are evaluated on 50 multi-turn tasks under 20 configurations with the official state checker; their score ranges are 24–40 percentage points (Table 2).

We additionally evaluated four models on the same 57 Terminal-Bench 1.0 tasks with four harness variants (912 task–model–variant cells). V1 uses baseline instructions with shell commands in Markdown code blocks and a standalone DONE completion marker. V2 paraphrases the instructions while keeping that protocol. V3 uses lowercase <think>, <cmd>, and <done/> tags; V4 changes only those tags to uppercase <THINK>, <CMD>, and <DONE/>.

ModelV1V2V3V4Std. (pp)Range (pp)
GPT-5.673.7%63.2%66.7%50.9%9.522.8
Claude-Sonnet36.8%49.1%15.8%12.3%17.536.8
Claude-Opus66.7%70.2%61.4%63.2%3.98.8
GLM-5.329.8%38.6%33.3%29.8%4.18.8

Accuracy is reported on the same 57 tasks; sample standard deviation and range summarize the four variant accuracies.

GPT-5.6 falls from 73.7% under V1 to 50.9% under V4, a 22.8-point difference. Thirteen tasks change from pass to fail and none change in the reverse direction. The effect appears in both task batches: 6:0 in the first and 7:0 in the second. GPT-5.6 also moves from above Opus under V1 to below it under V4. These results show that harness choice affects both success rates and model comparisons in terminal workflows.

The main HumanEval, BIRD, and MATH/AIME grid supplies the complementary controlled comparison: it spans frontier API and open-weight models while keeping tasks, tools, and budgets fixed as the harness changes. BFCL and Terminal-Bench then test the finding in multi-turn tool-use and terminal environments. Together, the experiments establish harness dependence across both controlled tasks and agent benchmarks.

AuthorsOpenReview ↗

Response to Reviewer qtm4 (3/3)

W2: Practical value of sensitivity auditing

The audit only ranks models by their harness sensitivity; it neither estimates accuracy nor recovers leaderboard rankings.

Accuracy and harness sensitivity answer two complementary questions: how well a model performs, and how much that performance changes with the evaluation interface. Our grid supports the usual accuracy summaries and leaderboard rankings through means or medians across configurations (Table 1). ScaffoldSpread adds the sensitivity dimension, measured as the standard deviation of accuracy across configurations.

The practical analogy is reporting both a mean and a standard deviation. Two models with similar average accuracy can behave very differently when a user changes the harness. A deployment that needs predictable behavior favors stability; a user willing to tune the harness may instead seek configurations that unlock higher performance. Reporting accuracy together with sensitivity makes this trade-off visible.

To make that check affordable, we use eight rather than all 32 configurations. Across the 72 model–domain combinations, the eight-configuration audit recovers the full-grid sensitivity ranking with Spearman ρ = 0.9812, using one quarter of the configuration budget. Selecting configurations on two domains and evaluating them on the third gives ρ = 0.919–0.968. This makes sensitivity reporting a practical addition to a leaderboard.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading and questions. Our central finding is that equivalent harness choices change measured model comparisons. The additional controlled comparisons and agent experiments address the main concerns about interface robustness, configuration outliers, and practical relevance.

Can a few problematic configurations explain the result? We removed each configuration in turn and recomputed code-domain ranking agreement. Even the single removal that improves agreement the most raises mean Kendall τ only from 0.686 to 0.697. Removing the three configurations with the largest average score deviations gives 0.696; the mean cross-configuration accuracy standard deviation remains 9.38 percentage points, compared with 10.34 in the full grid. The instability persists across the remaining configurations.

Does the result depend on changing interfaces or reminders? We excluded all native-FC configurations—those using native function/tool calling—and kept the turn reminder on throughout. This leaves 18 configurations that vary presentation without switching to the native interface or removing the reminder. Mean Kendall τ is 0.685 on code, compared with 0.686 in the full grid; the corresponding math and SQL values are 0.870 and 0.822. The code result is essentially unchanged under this stricter control.

What remains after tolerant extraction? Re-scoring the same saved code outputs after removing outer code fences and tags raises the pass rate from 78.6% to 86.3%. In the controlled native-call comparison with that tolerant extraction applied, 11 of 253 model pairs still reverse and six models change accuracy by more than 10 points. Thus, repairing extraction and measuring the remaining sensitivity are complementary parts of a reliable evaluation.

Comparison with arXiv:2608.21382. The cited work focuses on fixed-choice questions: which items change correctness under different harnesses? Our study examines open-ended generation, code, and multi-turn tool use, settings explicitly outside that paper’s scope. Here the harness governs a sequence of actions and feedback, so we study how that interaction changes model comparisons, how failures shift between delivery, extraction, and content, and how to audit the resulting sensitivity. This extends the evaluation question from selecting a fixed answer to completing a task through a protocol.

Evidence from agent benchmarks. The paper’s BFCL v3 experiment gives 24–40-percentage-point score ranges across harnesses (Table 2). In the additional Terminal-Bench study, four models are evaluated on the same 57 tasks under four harness variants. GPT-5.6 falls from 73.7% to 50.9% between two command-presentation formats: 13 tasks change from pass to fail, with no reverse changes. It also moves from above Opus to below it. These results extend the evidence to multi-turn tool use and terminal workflows.

Making sensitivity practical to report. ScaffoldSpread uses eight of the 32 configurations to recover the full-grid sensitivity ranking with Spearman ρ = 0.9812, at one quarter of the configuration budget. Reporting performance together with its variability across harnesses, much as a mean is reported alongside a standard deviation, helps readers distinguish consistently strong performance from an advantage tied to one harness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.