acceptodds
Under review as a conference paper at ICLR 2027

QuoteBench: When Matched Scores Hide Command-Path Failures

Abstract

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench separates the two by replay. It holds a model's reply fixed and executes it on two transports, direct and through one deliberately unescaped added parser, and crosses that with two generation contracts, one silent about the boundary and one that discloses it. The matched gap between the two paths then decomposes exactly, per task, into transport damage and contract-conditioned compensation. Across eight same-window configurations on 56 exact-final-state tasks from 14 incident-derived families, replaying the same reply through the added parser loses 55.4–73.2 points; disclosure recovers 30.4–60.7 points for six configurations and zero or slightly negative for the other two, and three public draws preserve both ranges. Raw generation is nearly saturated at the frontier, so boundary adaptation is what separates models: at the frontier the two effects cancel (GPT-5.6-sol's matched gap of −3.6 points hides −64.3 damage and +60.7 compensation), while elsewhere the matched score falls with the path, and the deployment configuration reorders models. A real ssh boundary reproduces the damage, and correct escaping or a temporary script removes it. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator instead of treating a matched score as an intrinsic model property.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, coding agents, command execution, evaluation methodology, execution transport, benchmark, shell quoting, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U7TtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how Bash commands generated by code agents can suffer from literal corruption when subjected to additional parsing during execution, and explains why high "success rates" in standard evaluation settings can mask this path dependency. The authors introduce "QuoteBench," a benchmark comprising 14 operation families and 56 tasks; each family includes a standard control task alongside three variants involving "dangerous" payloads. The tasks span a range of contexts, including file content, filenames, command arguments, environment variables, JSON, Git, regular expressions, and nested execution (such as via SSH). Evaluation is based on verifying the final state—such as file byte content, argument vectors, or other target system states. The core experiments involve a cross-combination of "generation contracts" (the assumptions under which a model generates code) and "execution paths" (the actual environments in which that code is executed). Models generate responses under two different contracts—one assuming direct execution and another where the boundaries of additional parsing are explicitly disclosed—and each response is then tested across multiple execution paths. This allows the performance gap under different conditions to be decomposed into two factors: performance degradation caused by the execution path itself, and performance shifts resulting from changes to the generation contract while keeping the execution path constant. Across eight configurations, the introduction of additional, unescaped parsing causes success rates to drop by 55.4 to 73.2 percentage points. While six configurations demonstrate an ability to adapt their output based on disclosed parsing boundaries, two show no such positive compensation. The authors further explore this through repeated sampling, private payloads, varied user environments, real-world SSH scenarios, comparisons between single- and double-quote syntax, and benchmarks against properly escaped code or temporary script files. The primary contribution of this paper is an evaluative framework that distinguishes between failures originating in the command generation process and those introduced by the execution path, demonstrating that a high success rate in a specific, matched environment can coexist with poor portability across different execution paths.

Critical strengths
  1. The use of "fixed-output replay" provides a clear mechanism for identification, effectively ruling out major confounding factors introduced by regeneration. In my view, the most significant design choice in this paper is the decision to route the same model-generated response through different execution paths, rather than comparing final scores across entirely different agent frameworks. This ensures that variations in outcomes for a given response cannot be attributed to differences in re-planning, tool selection, or the inherent stochasticity of the generation process. Out of the 448 model-task combinations presented in Table 1, the original execution path yielded 415 successes; however, 292 of these became failures once parsing boundaries were introduced. Both "correct escaping" and the use of temporary scripts restored all of these original successes, while failing to "fix" commands that were already unsuccessful. This set of controls provides strong evidence that the observed discrepancies stem specifically from the intervened execution paths. The scientific value lies in the empirical measurement of controlled, counterfactual execution outcomes. While the decomposition formula itself is an identity, the replay design imbues each of its components with a precise, verifiable experimental meaning.
  2. The paper provides a robust validation against key alternative explanations, ensuring that the primary findings are not merely artifacts specific to a single "wrapper" or framework. Repeated sampling demonstrates that the direction of execution impairment remains consistent across all configurations and sampling iterations. Tests using private payloads further reduce the likelihood that the results depend solely on publicly available literal strings. Moreover, replays involving actual SSH sessions confirm that this parsing-based mechanism manifests in real-world remote command execution. The cross-experiment involving single versus double-quote syntax is particularly insightful. Stronger models perform significantly better when the syntax disclosed to them matches the syntax used for actual execution—indicating that they actively adjust their output based on the specified boundaries. This finding helps distinguish between "contractual adaptation" (adhering to specified constraints) and general command robustness. The validator was also subjected to rigorous checks, including evaluations against original states, correct solutions, intentionally flawed solutions, and 197 state variations. Although the handling of "additional file" checks could be clarified further, this auditing approach significantly enhances the interpretability of the benchmark results.
Critical weaknesses

W1. The most significant performance degradation observed in the paper stems from a specific, intentionally introduced execution path: inserting a complete program—originally intended for direct execution—into a double-quoted wrapper without proper escaping. While this experiment reliably identifies the damage caused by such a transformation, assessing its practical impact on existing agent system evaluations would require a more direct link to the command-processing mechanisms found in real-world systems.

Appendix A.1.1 examines 37 command paths across 24 systems. Only one is classified as involving an unescaped secondary parse—specifically, the path used by Aider on Windows via cmd.exe and PowerShell, which falls outside the scope of the current POSIX/Bash-based benchmark. Most paths involve only a single parsing stage, while five others entail byte-level modifications—mechanisms that differ from the primary intervention explored in this paper.

I note that the authors explicitly define "nested" scenarios as controlled stress conditions and make no claims regarding the frequency of such occurrences in actual deployments. Real-world SSH replay scenarios and five publicly documented incidents involving secondary parsers do support the plausibility of these mechanisms. However, since the SSH experiments also involve the deliberate construction of the same unescaped wrapper, they primarily demonstrate semantic consistency rather than providing direct evidence of the specific failure mode under investigation within the agent paths studied.

This limitation affects the assessment of the paper's practical significance. While the current evidence effectively supports a diagnostic benchmark targeting a specific failure boundary, it is insufficient to establish the benchmark's predictive or diagnostic value regarding actual failures in common agent frameworks. The argument could be strengthened by replaying portions of the saved model responses directly into the command-transformation pipelines of the systems under study, or by more precisely framing the contribution as a "boundary stress test."

W2. In Section 4.2, the paper describes the reversal in rankings between GPT-5.6-sol and Gemini-3.5-Flash as clear at the current resolution. According to Table 1, the two models succeed on 53 and 54 tasks, respectively, under the "RR" metric, and on 51 and 33 tasks, respectively, under the "NN" metric. Therefore, a reversal observed on one side may hinge on the performance gap in a single task; a substantial lead on the other side is insufficient, on its own, to establish that the ordering along the original path is robust.

Table 2 offers a more direct constraint. Based on the mean success rate across three sampling runs, GPT-5.6-sol achieves an RR of 97.0% and Gemini-3.5-Flash 96.4%, while the NN baselines reach 94.6% and 51.2%, respectively. This implies that, when evaluated by mean success rate, both models maintain the same relative ordering across the two conditions. Appendix E.1.6 presents reversals based on the number of tasks passed in all three sampling runs; however, this metric differs from the mean success rate and must be clearly distinguished.

The paper reports Kendall’s tau correlation coefficients, bootstrap confidence intervals, and partial rankings—analyses that support the conclusion that overall rankings are path-dependent. Nevertheless, these do not automatically demonstrate that, for any specific pair of models, the observed reversal in ordering across the two paths is statistically robust. It is necessary to provide paired difference intervals and a measure of joint reversal stability for such model pairs, while also clarifying whether resampling at the task level versus the model-family level yields consistent conclusions.

While this issue does not invalidate the finding of significant performance degradation, it weakens the specific argument that the choice of execution path fundamentally alters which model performs better. Reversals should be reported separately for single-sample outcomes, mean performance, and metrics of robust execution.

Regarding W3: Section 3.2 states that the execution process includes checks for the presence of extraneous files, and the examples in Table 3 likewise require the absence of such files. Appendix F.1 further notes that all 56 test variants—in which unexpected files were introduced into an otherwise correct state—were rejected by the validator. According to these descriptions, the creation of extraneous files constitutes a failure.

However, the "Limitations" section on page 9 explicitly states that the presence of extraneous files and issues related to file permissions are not factored into the scoring. These two descriptions imply different criteria for correctness. If only the target files are inspected, a command might be considered successful even if it produces unintended side effects; conversely, if the entire working directory is evaluated, the same command might be deemed a failure.

This discrepancy also has implications for comparisons involving temporary scripts. The paper reports results obtained by directly executing scripts located in temporary directories; therefore, it must be clarified whether these auxiliary scripts reside outside the directory under evaluation, whether they are removed prior to verification, and whether this procedure is applied consistently across all experimental batches.

The available materials are insufficient to conclude that the actual implementation is flawed, and the audit presented in Appendix F.1 provides some positive evidence supporting a rigorous evaluation. However, given that the verification of the final state serves as the core basis for the entire benchmark, the authors should provide a unified definition of "success" and clarify whether the scope of verification varies across different experimental campaigns.

Questions for authors
  1. Which execution paths in the actual system correspond most directly to the primary interventions? In Table 8, an unescaped "second parsing path" is identified within the Windows environment; however, the current benchmark excludes this environment. Have any POSIX/Bash-based paths been investigated that would allow for the reproduction of these responses using the actual command-processing code? Furthermore, if the target is primarily an SSH wrapper constructed by the model itself, is it possible to distinguish between an erroneous wrapper written by the model and an error introduced by the framework during post-generation processing?

  2. What are the statistical criteria for defining a "rank reversal"? In Table 2, the average success rates for GPT-5.6-sol and Gemini-3.5-Flash no longer show a reversal; why, then, is this pair still characterized as exhibiting a clear rank reversal? Could you provide the average success rates, the "all-three-trials-passed" rates, and the confidence intervals for the paired differences across the two paths? Specifically, what stability criteria define the "22 stable pairwise orderings"?

  3. Are changes to auxiliary files factored into the final evaluation across all experiments? How can the descriptions in the "Limitations" section and Appendix F.1 be reconciled? Do all experimental campaigns employ the same criteria for directory inspection? Additionally, where are temporary scripts created, and at what point are they cleaned up—could these processes affect the verification results?

What specific types of failures are actually resolved through "model adaptation"? Beyond the average compensation magnitude, could you report the number of tasks that follow an "RR (success) → RN (failure) → NN (success)" pattern, as well as the number of new failures that emerge following a change in the contract? Such task-level transitions would clarify whether the compensation primarily restores functionality lost on the original path, or if it also reflects shifts in the set of successful outcomes resulting from the generation of different commands.

4 comments

Thank you for the detailed review. W2/Q2 and W3/Q3 are answered here; W1/Q1 and Q4 in the second part.

W2 / Q2. Statistical criteria for the rank reversal.

We removed "unambiguous" and defined reversal; the robust finding is the separation on the nested path (9 of 15 intervals exclude zero, 0 of 15 on the raw path). A pair reverses when its point estimates take strictly opposite orders on the two paths; ties are not comparable; counts are descriptive. → Section 4.2, App. E.1.7

GPT-5.6-sol vs Gemini-3.5-Flash, separately for each reading you asked for:

ReadingRaw pathNested pathReversed?
Single draws2 of 3 draws
Three-draw means97.0 vs 96.4; difference +0.6 [−3.0, +3.6]94.6 vs 51.2; difference +43.5 [+27.4, +60.1]No
Tasks passed in all three draws53 vs 5150 vs 18No
RequestAddedWhere
Paired difference intervalsAll 15 pairs among the 6 compensating configurationsTable 30
Joint reversal stabilityJoint flip rate per pairTable 30
Task vs family resamplingBoth reportedTable 30
Draws resampled as well9 of 15 nested-path intervals exclude zero, 0 of 15 raw-pathTable 30
Single / mean / all-three reportingSeparate counts under the three rules; the three pairs that reverse under all three differ by ≤ 1.8 raw points, and no pair reverses in every single drawApp. E.1.7
"22 stable orderings"Trial-0 pairs keeping one strict order in ≥ 95% of task resamples: 22 of 28 nested, 14 of 28 rawApp. E.1.7
Rank correlationKendall τa = 0.57App. E.1.7

W3 / Q3. One definition of success; temporary scripts; consistency across campaigns.

Section 3.1 now states one rule, and the Limitations sentence matches it. → Section 3.1

ItemRule
Target stateChecked exactly by the task's validator
New file outside the task's allowed setRun fails
Edits or deletions of other existing filesCounted where the validator compares the whole directory (8 of the 40 tasks with such files)
Privilege effectsNot scored
Temporary script.qb-transport-script.sh, written in the working directory, deleted before validation, never counted as a new file; script replays match the raw path on all 448 public and 126 private pairs (App. E.1.1)

Which campaigns apply the new-file check → App. F.1

CampaignEvidence
Every campaign except the live BSD runsSame success rule, new-file check included
Real-ssh replaysRerun with the check: no reply rejected, no count changed (Tables 7, 28)
Replays rerun with file listsTable 1 (1,792 runs), JSON boundary, Study B local (8,736), Sonnet-4.6 crossover (224): every stored outcome reproduces
All public-task records with file lists0 of 46,235 passes leave a disallowed file
Live BSD runsPredate the check; the GNU replay of the same replies applies it (Table 17 caption)

W1 / Q1. A direct link to real systems' command processing.

The revision does both remedies you proposed.

Your remedyWhat we didWhere
Replay saved responses through real command-transformation pipelines448 stored replies through 10 open systems' command-handling code at the surveyed commits, in our container with our validators (component replay, zero model calls)App. A.1.2, Table 9
Frame as a boundary stress testAbstract, Contribution 1 and Limitations now say soAbstract, Sections 1, 5

Real-code replay results:

SystemsResult
Codex, Goose, mini-swe-agent, Desktop Commander (single shell)Match the raw path on all 448
The other systemsDiffer only by their own tokenization, shell selection, filters and session handling; never by a second parse

Which real path corresponds most directly? A wrapper the agent writes, e.g. ssh host "..." through Codex. Codex passes the command as one shell -c argument (Table 5), like our raw path; the remote shell parses it again under our nested rules. Such wrappers are common: 27,096 of 53,938 Bash commands in our session logs go through ssh (App. A.1). Agent-written results → App. E.1.5:

SettingResult
Two SSH-like families, all 8 configurations58.3–100% pass
12-family pilotFlash-Lite 50.0% vs 83.3% raw; Sonnet-4.6 67.4% vs 85.4%; Opus-5 and Opus-4.8 lose nothing

The real-ssh replay constructs the same wrapper (your W1). That is by design: it tests that the nested path behaves like a real remote command (893 of 896 replies, Tables 7, 28). The case where the model builds the wrapper itself is the agent-written table above.

Model's wrapper or the framework? Rerun the stored command on the raw path with task, environment and validator fixed. A failure there predates any harness transformation; a raw-path pass that fails on the deployed path isolates damage added by the path.

Q4. Task-level transitions.

Compensation mostly restores what the path broke. Trial 0, 448 model–task pairs → Section 4.3, Table 20

RR → RN → NNCount
Pass → fail → pass (restored)164 of 292
Pass → fail → fail128
Pass → pass → pass113
Pass → pass → fail (new failures)10 of 123 (2.2% of 448)
Fail → fail → pass9
Fail → fail → fail24
  • Explicit hazards recover: json-write (damage −90.6, compensation +50.0), sed-replace (−43.8, +46.9).
  • Implicit hazards stay broken: find-glob, grep-count, hostile-filenames.
  • Cost on the raw path: across the six compensating configurations, RR 323 → NR 175 passes; 154 RR passes fail as NR and 6 RR failures pass.

Thank you for addressing the statistical and validator concerns. My remaining concern is whether the narrowed contribution provides sufficient insight beyond the known effects of unescaped parsing. To consider raising my score to 3. I need evidence that QuoteBench reveals reproducible failure patterns with practical diagnostic value. The current pilot and aggregate compensation counts do not yet yet establish that connection. I therefore retain 2 score for now.

Thank you for the follow-up. We added a zero-call analysis of the stored replies (App. E.1.7, "Per-task diagnosis"). It shows that QuoteBench gives every model–task pair a diagnosis that repeats across draws, names the fix, and locates failures on wrappers the model writes itself.

1. Beyond unescaped parsing. Unescaped parsing predicts only that a reply breaks on the wrapper, and the wrapper leaves a similar share passing for every configuration (RN 19.6–30.4%, Table 1). The four cells add what that effect cannot: whether this model repairs the reply once told the path.

DiagnosisCellsFix
Generation failureRR failChange the model
Survives the wrapperRN passNone
CompensatedRN fail, NN passDisclose the path
UncompensatedRN fail, NN failFix the path; a temporary script recovers every reply the wrapper breaks (292 of 292 in trial 0, App. E.1.1)

2. Reproducible. Diagnosis identical in all three independent draws:

ConfigurationSame diagnosis (of 56)Fleiss κSame compensated/uncompensated split (damaged in all draws)
GPT-5.6-sol470.8030/35
GPT-5.5410.6726/38
Opus-5440.7327/35
Gemini-3.1-Pro470.8134/40
Gemini-3.5-Flash310.5421/34
Opus-4.8440.7829/33
Qwen3.5-27B550.9830/30
Gemini-3.1-Flash-Lite530.9432/32
All362/448 (81%)229/277 (83%)

3. Diagnostic value: task-level transfer to agent-written wrappers. The model writes its own ssh-like call, with the same payload and validator (App. E.1.5). This tests each of the 133 failed runs, beyond the four-configuration ranking:

ConfigurationFailed runsOn tasks where the same model failed the disclosed nested pathExpected if random
Opus-5330.7
Opus-4.811117.1
Gemini-3.1-Flash-Lite727261.5
Sonnet-4.6 (one nested draw)473833.3
All133124102.6 (one-sided permutation p < 0.001)

Practical use. One QuoteBench run tells a team, per operation, whether to change the model, disclose the path, or fix the path. The answer repeats across draws for 81% of model–task pairs, and a model's disclosed-path failures mark where its own wrappers fail. We hope this establishes the connection you asked for.

Reviewer_sFVhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

QuoteBench evaluates how execution paths affect the reliability of LLM-generated Bash commands. It contains 56 exact-final-state tasks across 14 operation families. The central design crosses generation contracts with execution transports and replays each stored output through both direct execution and an additional unescaped shell boundary. This distinguishes command-generation errors, fixed-output transport damage, and contract-conditioned adaptation. Across eight same-window configurations, the added boundary reduces success by 55.4–73.2 percentage points; disclosing the boundary substantially compensates for this loss in six configurations. Repeated generations, private payloads, grammar crossover, and real-SSH replay support the mechanism. Correct escaping and temporary scripts eliminate the measured transport damage in the reported controls.

Critical strengths
  1. The crossed replay design provides clean attribution. Executing the same reply through different transports distinguishes generation failures from errors introduced downstream. The GPT-5.6-sol example illustrates why this matters: a −3.6-point matched gap conceals −64.3 points of transport damage and +60.7 points of compensation. Although the decomposition is algebraically simple, the experimental design makes its components observable.
  2. The mechanism is supported by substantial controls. Real-SSH replay, private payloads, repeated draws, and single-/double-quote grammar crossover strengthen the interpretation. Correct escaping and temporary-script execution reproduce direct-path outcomes, directly identifying the source of the damage. Validator mutation audits further support the reliability of the narrow benchmark.
Critical weaknesses
  1. Limited significance beyond a repairable interface defect. The main transport loss follows from unsafe shell interpolation, and correct escaping eliminates it in the reported controls. The additional research value of measuring model compensation remains insufficiently established.

  2. A gap between the deployment survey and the tested intervention. In the extended survey, the only unescaped second-parse path is a Windows/PowerShell case, outside the benchmark’s stated scope. Real-SSH replay validates the mechanism, but does not establish its impact in the surveyed Bash-agent deployments.

  3. The decomposition is reference-path dependent. The reported damage and compensation terms use RN as the intermediate condition. With substantial contract–transport interaction, these are useful reference-specific contrasts rather than a unique decomposition into independent mechanisms.

  4. Transfer beyond the tested boundary family remains unclear. Private payloads and related quote grammars support within-setting generalization. The JSON experiment demonstrates transport damage, but does not test whether contract-conditioned adaptation transfers beyond shell wrappers.

Questions for authors
  1. Can you provide a concrete deployment case where the crossed diagnosis changes a meaningful evaluation or system-design decision, beyond identifying the need for correct serialization?
  2. Once correct boundary encoding removes the measured damage, what is the intended continuing role of the compensation score: model selection, contract-adherence evaluation, or interface regression testing?
  3. Is compensation on the tested shell wrappers predictive of performance on a held-out execution boundary? If this has not been evaluated, would you explicitly restrict the capability interpretation accordingly?
2 comments

Thank you for the careful review. W1–W4 are answered here and Q1–Q3 in the second part.

W1. Limited significance beyond a repairable interface defect.

Compensation matters where no harness can escape the command for the model: wrappers the agent writes inside its own command, such as ssh host '...'. A general harness receives the finished command and cannot reliably repair every inner boundary by escaping it once. → Section 1

Where the caller builds the wrapper, caller-side escaping is the fix, and the paper recommends it first. → Section 5

Agent-written wrappersResultWhere
Two SSH-like families, all 8 configurations58.3–100% pass; an appended wrapper gives 0%App. E.1.5
Other 12 families, 4-configuration pilotLow-compensation configurations lose 18–33 points (Q2)App. E.1.5

W2. Gap between the deployment survey and the intervention.

A code survey sees only wrappers the harness adds; the wrappers we target are written by the agent inside its command. Evidence on those:

EvidenceResultWhere
Session logs27,647 of 53,938 Bash commands cross a second parser, 27,096 via sshApp. A.1
Incidents≥ 63 of 86 fail across a wrapper; in the 50 Codex incidents, agents retried in 27, changed strategy in 12, altered the task text in 5, gave up or degraded the task in 4App. A
Real command-handling code of 10 open systemsThe four single-shell systems match the raw path on all 448 replies; no difference comes from a second parseApp. A.1.2, Table 9
Real sshSame outcome as the nested path on 893 of 896 repliesTables 7, 28

W3. The decomposition depends on the reference path.

The revision reports both splits; large opposing terms appear in both. → Section 4.2, Table 19

GPT-5.6-sol, trial 0Path effectPrompt effect
Split through RN−64.3 (damage, RN − RR)+60.7 (compensation, NN − RN)
Split through NR+35.7 (NN − NR)−39.3 (NR − RR)

Four configurations show large opposing terms in both splits; their sizes depend on the split.

W4. Transfer beyond the tested boundary family.

The adaptation claim is now restricted to shell wrappers; the JSON experiment shows damage only. Evidence within shell wrappers is under Q3. → Sections 1, 5

Q1. A deployment case where the crossed diagnosis changes a decision?

The crossed cells change what a team fixes; the deployed-path score still picks the model. Two decision cases from our controlled data → Sections 4.3, 5

CaseOne score aloneCrossed cells
GPT-5.6-sol vs Qwen3.5-27BBoth pass 30.4% of raw-prompt replies on the nested path (RN)Disclosure lifts GPT-5.6-sol to 91.1%, Qwen3.5-27B stays at 30.4%: fix the prompt for one, the path or model for the other
Opus-4.8, effort low → maxMatched gap −48.2 → −3.6, as if the path problem went awayRN stays at 28.6–33.9% across its five settings while NN rises from 39.3 to 94.6: the gain is the disclosed prompt (all six laddered configurations: RN 23.2–33.9%, ≤ 5.4 points within a ladder)

Q2. The continuing role of compensation?

Contract-adherence evaluation: whether a model, told a boundary, writes a command that survives it. The new pilot tests it on wrappers the model writes itself (12 families, 3 draws) → App. E.1.5

ConfigurationCompensationAgent-written passRaw path, same 48 tasksLoss
Opus-5+61.397.997.90
Opus-4.8+23.292.492.40
Sonnet-4.6 (medium)+1.8 (one draw)67.485.4−18.1
Gemini-3.1-Flash-Lite−6.550.083.3−33.3
  • The agent-written rate follows the compensation ranking (Spearman ρ = 1.0, exact one-sided p = 0.042).
  • Losses exceed raw-path gaps: raw-path rates span 14.6 points, agent-written rates 47.9. Sonnet-4.6 trails Opus-4.8 by 7.0 raw points but loses 18.1 on its own wrapper; their compensation differs by 21.4.

Your three options:

RoleOur answerWhere
Model selectionUse the matched score on the deployed pathSection 5
Contract-adherence evaluationYes: the pilot aboveApp. E.1.5
Interface regression testingThe crossed replay does this: stored replies replayed on the new path show whether they survive the switchSection 5

Q3. Held-out boundary: will you restrict the capability interpretation?

Yes. The paper now restricts it to shell wrappers, the boundary on which we measured adaptation. → Sections 1, 5

Within shell wrappers, two further settings keep the ordering of configurations: agent-written wrappers follow the compensation ranking (Q2), and on a second grammar replies do not carry over but the grouping of configurations does (tasks passed of 56) → App. E.1.3, Table 27

ConfigurationsTold double, run doubleTold double, run singleTold single, run single
GPT-5.6-sol, GPT-5.5, Opus-549–547–845–54
Sonnet-4.6, Opus-4.8, Haiku-4.525–3211–1320–23
Qwen3.5-27B, Gemini-3.1-Flash-Lite7–1711–199–11
Reviewer_PQJPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

I had a hard time understanding the details of this paper despite my best attempts because of AI writing. There are many unusual terms ot describe the setting that I did not understand. I did not use AI to try and understand this paper because I believe papers should be written for a human audience.

The paper is trying to understand the success/failures of AI agents and how it depends on the execution environment. They show that many agents achieve similar scores in general but perform quite differently when something about the execution infrastructure changes.

Critical strengths

Paper seems to study an important nuance and I trust that the execution/results are correct as they seem straightforward.

Critical weaknesses

I couldn't understand the details of this paper, what's the exact problem studied. They introduce a new parser as part of the execution framework but I don't understand what this parser is, and why introduce this specific modification - is this supposed to be practically relevant or a scientific tool? What about other modifications one could make? The paper says that they identify the specific mechanism - is this a general mechanism or specific to their parser?

Questions for authors

See weaknesses above

1 comment

Thank you for identifying the clarity problems. We now open with a worked example and replace the unusual terms. Each point is answered below.

Writing and unusual terms. The main text is rewritten around one worked example.

ChangeWhere
"Generation contract" → "prompt", "execution transport" → "path"throughout
Opening example (price: $5), Figure 1 moved to p. 2pp. 1–2
Each term defined where it first appearsSections 1–3
One task traced through all four cells with real repliesApp. C.2, Table 18

Q1. What exact problem is studied?

When a command fails, a score on one path cannot tell whether the model's command or the path caused it, and a matched score can hide both. → Abstract, pp. 1–2

Q2. What is the parser?

One extra round of shell parsing. The worked example that now opens the paper → p. 1:

The same commandResult
echo "price: \$5" > out.txt, run directlywrites price: $5
inside bash -c "..."exits 0, prints price:, creates no file
over a real ssh host "..."creates the file without the price

Inside double quotes the outer shell still expands $, backticks and backslashes, so ordinary text breaks: even the 14 benign control tasks lose 28.6–57.1 points. Table 18 shows the same effect on real model replies. → Section 4.2, App. C.2

Q3. Why this modification? Practical or a scientific tool?

A scientific tool for one boundary that is common in practice. The paper now calls it a stress test of this boundary. → Abstract, Section 1

EvidenceResultWhere
Controlled: it changes one variableSame reply, same validator; only the added parser differsSection 1
Matches a real boundaryReal ssh gives the same pass/fail as the nested path on 893 of 896 repliesTables 7, 28
Common in practice27,647 of 53,938 Bash commands in our session logs cross a second parserApp. A.1
Fails in practiceAt least 63 of our 86 internal incidents fail across a wrapperSection 1, App. A.1

Q4. What about other modifications?

We tested four on the same stored replies.

ModificationEffectWhere
Single-quoted wrapper (the most common form in our logs)−48.2 to −85.7 points, 8 configurationsApp. E.1.3
Naive JSON embedding−51.8 to −66.1 points, 6 configurations; 34–47 of 56 replies fail to parseApp. E.1.6
Correct escaping by the callerRestores every raw-path successSection 5
Temporary scriptMatches the raw path on all 448 public and 126 private pairsApp. E.1.1

Q5. Is the mechanism general or specific to this parser?

The mechanism is general: a second parser reading text meant for the first consumes its characters. The measured sizes hold for the three boundaries tested, and model adaptation is shown for shell wrappers; the paper now states both. → Sections 1, 5

Comments on the paper

1 comment

We thank the three reviewers. The revised PDF is uploaded; blue marks substantive changes. Each reviewer's points are answered in the individual replies.

Main results (unchanged). Replaying the same reply through one added parser costs 55.4–73.2 points across eight configurations; disclosing the parser recovers 30.4–60.7 points for six of them (Table 1). A real ssh command matches the nested path on 893 of 896 stored replies (Tables 7, 28). Caller-side escaping and a temporary script restore every raw-path success (Section 5, App. E.1.1).

New experiments and checks

ExperimentResultWhere
Stored replies through ten open systems' command-handling code (448 replies)The four single-shell systems match our raw path on all 448; no difference comes from a second parseApp. A.1.2, Table 9
Agent-written wrappers, two SSH-like families, all 8 configurations58.3–100% pass when the model writes the second parser; an appended wrapper gives 0%App. E.1.5
Agent-written pilot, other 12 families, 4 configurations × 3 draws (576 calls)Pass rate follows the compensation ranking (Spearman ρ = 1.0, p = 0.042); low-compensation configurations lose 18–33 pointsApp. E.1.5
Task-level transitions, 448 model–task pairsOf 292 passes broken by the wrapper, the disclosed prompt restores 164; it breaks 10 of 123 survivorsSection 4.3, Table 20
Pairwise intervals and joint flip rates, 15 pairs, family and task resampling, draws resampledIntervals exclude zero for 9 of 15 nested-path differences and 0 of 15 raw-path onesSection 4.2, Table 30
Session-log count, 53,938 Bash commands27,647 cross a second parser, 27,096 of them via sshSection 1, App. A.1
Table 1 replay rerun with per-run file lists (1,792 runs)All outcomes reproduce; no pass leaves a file outside the allowed setApp. F.1
JSON replay rerun with file listsIdentical results; 0 of 719 passes leave a disallowed fileApp. F.1
Real-ssh replays rerun with the new-file checkNo reply rejected, no count changedTables 7, 28
One task in all four cells with real model repliesShows each cell's command and outcomeApp. C.2, Table 18
Same-window Sonnet-4.6 crossover (medium effort)RR 87.5, RN 26.8, NR 82.1, NN 28.6Table 15

Revisions

PointRaised byChangeWhere
Rank reversal of GPT-5.6-sol and Gemini-3.5-FlashU7Tt W2Reversal defined and counted under three rules; the paper now leads with the separation on the nested path (9 of 15 intervals exclude zero, 0 of 15 on the raw path)Section 4.2, App. E.1.7
Scoring of extra filesU7Tt W3One scoring rule in Section 3.1; Limitations matches itSection 3.1
Reference path of the decompositionsFVh W3Both splits reportedSection 4.2, Table 19
Scope of the adaptation readingsFVh W4, Q3Stated for the tested shell wrappersSections 1, 5
Practical scopePQJP, sFVh W2, U7Tt W1Stated as a stress test of the second-parser boundaryAbstract, Sections 1, 5
Unusual termsPQJP"Generation contract" → "prompt", "execution transport" → "path"; worked example opens the paper; Figure 1 on p. 2Sections 1–3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.