- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how Bash commands generated by code agents can suffer from literal corruption when subjected to additional parsing during execution, and explains why high "success rates" in standard evaluation settings can mask this path dependency. The authors introduce "QuoteBench," a benchmark comprising 14 operation families and 56 tasks; each family includes a standard control task alongside three variants involving "dangerous" payloads. The tasks span a range of contexts, including file content, filenames, command arguments, environment variables, JSON, Git, regular expressions, and nested execution (such as via SSH). Evaluation is based on verifying the final state—such as file byte content, argument vectors, or other target system states. The core experiments involve a cross-combination of "generation contracts" (the assumptions under which a model generates code) and "execution paths" (the actual environments in which that code is executed). Models generate responses under two different contracts—one assuming direct execution and another where the boundaries of additional parsing are explicitly disclosed—and each response is then tested across multiple execution paths. This allows the performance gap under different conditions to be decomposed into two factors: performance degradation caused by the execution path itself, and performance shifts resulting from changes to the generation contract while keeping the execution path constant. Across eight configurations, the introduction of additional, unescaped parsing causes success rates to drop by 55.4 to 73.2 percentage points. While six configurations demonstrate an ability to adapt their output based on disclosed parsing boundaries, two show no such positive compensation. The authors further explore this through repeated sampling, private payloads, varied user environments, real-world SSH scenarios, comparisons between single- and double-quote syntax, and benchmarks against properly escaped code or temporary script files. The primary contribution of this paper is an evaluative framework that distinguishes between failures originating in the command generation process and those introduced by the execution path, demonstrating that a high success rate in a specific, matched environment can coexist with poor portability across different execution paths.
Critical strengths
- The use of "fixed-output replay" provides a clear mechanism for identification, effectively ruling out major confounding factors introduced by regeneration. In my view, the most significant design choice in this paper is the decision to route the same model-generated response through different execution paths, rather than comparing final scores across entirely different agent frameworks. This ensures that variations in outcomes for a given response cannot be attributed to differences in re-planning, tool selection, or the inherent stochasticity of the generation process. Out of the 448 model-task combinations presented in Table 1, the original execution path yielded 415 successes; however, 292 of these became failures once parsing boundaries were introduced. Both "correct escaping" and the use of temporary scripts restored all of these original successes, while failing to "fix" commands that were already unsuccessful. This set of controls provides strong evidence that the observed discrepancies stem specifically from the intervened execution paths. The scientific value lies in the empirical measurement of controlled, counterfactual execution outcomes. While the decomposition formula itself is an identity, the replay design imbues each of its components with a precise, verifiable experimental meaning.
- The paper provides a robust validation against key alternative explanations, ensuring that the primary findings are not merely artifacts specific to a single "wrapper" or framework. Repeated sampling demonstrates that the direction of execution impairment remains consistent across all configurations and sampling iterations. Tests using private payloads further reduce the likelihood that the results depend solely on publicly available literal strings. Moreover, replays involving actual SSH sessions confirm that this parsing-based mechanism manifests in real-world remote command execution. The cross-experiment involving single versus double-quote syntax is particularly insightful. Stronger models perform significantly better when the syntax disclosed to them matches the syntax used for actual execution—indicating that they actively adjust their output based on the specified boundaries. This finding helps distinguish between "contractual adaptation" (adhering to specified constraints) and general command robustness. The validator was also subjected to rigorous checks, including evaluations against original states, correct solutions, intentionally flawed solutions, and 197 state variations. Although the handling of "additional file" checks could be clarified further, this auditing approach significantly enhances the interpretability of the benchmark results.
Critical weaknesses
W1. The most significant performance degradation observed in the paper stems from a specific, intentionally introduced execution path: inserting a complete program—originally intended for direct execution—into a double-quoted wrapper without proper escaping. While this experiment reliably identifies the damage caused by such a transformation, assessing its practical impact on existing agent system evaluations would require a more direct link to the command-processing mechanisms found in real-world systems.
Appendix A.1.1 examines 37 command paths across 24 systems. Only one is classified as involving an unescaped secondary parse—specifically, the path used by Aider on Windows via cmd.exe and PowerShell, which falls outside the scope of the current POSIX/Bash-based benchmark. Most paths involve only a single parsing stage, while five others entail byte-level modifications—mechanisms that differ from the primary intervention explored in this paper.
I note that the authors explicitly define "nested" scenarios as controlled stress conditions and make no claims regarding the frequency of such occurrences in actual deployments. Real-world SSH replay scenarios and five publicly documented incidents involving secondary parsers do support the plausibility of these mechanisms. However, since the SSH experiments also involve the deliberate construction of the same unescaped wrapper, they primarily demonstrate semantic consistency rather than providing direct evidence of the specific failure mode under investigation within the agent paths studied.
This limitation affects the assessment of the paper's practical significance. While the current evidence effectively supports a diagnostic benchmark targeting a specific failure boundary, it is insufficient to establish the benchmark's predictive or diagnostic value regarding actual failures in common agent frameworks. The argument could be strengthened by replaying portions of the saved model responses directly into the command-transformation pipelines of the systems under study, or by more precisely framing the contribution as a "boundary stress test."
W2. In Section 4.2, the paper describes the reversal in rankings between GPT-5.6-sol and Gemini-3.5-Flash as clear at the current resolution. According to Table 1, the two models succeed on 53 and 54 tasks, respectively, under the "RR" metric, and on 51 and 33 tasks, respectively, under the "NN" metric. Therefore, a reversal observed on one side may hinge on the performance gap in a single task; a substantial lead on the other side is insufficient, on its own, to establish that the ordering along the original path is robust.
Table 2 offers a more direct constraint. Based on the mean success rate across three sampling runs, GPT-5.6-sol achieves an RR of 97.0% and Gemini-3.5-Flash 96.4%, while the NN baselines reach 94.6% and 51.2%, respectively. This implies that, when evaluated by mean success rate, both models maintain the same relative ordering across the two conditions. Appendix E.1.6 presents reversals based on the number of tasks passed in all three sampling runs; however, this metric differs from the mean success rate and must be clearly distinguished.
The paper reports Kendall’s tau correlation coefficients, bootstrap confidence intervals, and partial rankings—analyses that support the conclusion that overall rankings are path-dependent. Nevertheless, these do not automatically demonstrate that, for any specific pair of models, the observed reversal in ordering across the two paths is statistically robust. It is necessary to provide paired difference intervals and a measure of joint reversal stability for such model pairs, while also clarifying whether resampling at the task level versus the model-family level yields consistent conclusions.
While this issue does not invalidate the finding of significant performance degradation, it weakens the specific argument that the choice of execution path fundamentally alters which model performs better. Reversals should be reported separately for single-sample outcomes, mean performance, and metrics of robust execution.
Regarding W3: Section 3.2 states that the execution process includes checks for the presence of extraneous files, and the examples in Table 3 likewise require the absence of such files. Appendix F.1 further notes that all 56 test variants—in which unexpected files were introduced into an otherwise correct state—were rejected by the validator. According to these descriptions, the creation of extraneous files constitutes a failure.
However, the "Limitations" section on page 9 explicitly states that the presence of extraneous files and issues related to file permissions are not factored into the scoring. These two descriptions imply different criteria for correctness. If only the target files are inspected, a command might be considered successful even if it produces unintended side effects; conversely, if the entire working directory is evaluated, the same command might be deemed a failure.
This discrepancy also has implications for comparisons involving temporary scripts. The paper reports results obtained by directly executing scripts located in temporary directories; therefore, it must be clarified whether these auxiliary scripts reside outside the directory under evaluation, whether they are removed prior to verification, and whether this procedure is applied consistently across all experimental batches.
The available materials are insufficient to conclude that the actual implementation is flawed, and the audit presented in Appendix F.1 provides some positive evidence supporting a rigorous evaluation. However, given that the verification of the final state serves as the core basis for the entire benchmark, the authors should provide a unified definition of "success" and clarify whether the scope of verification varies across different experimental campaigns.
Questions for authors
-
Which execution paths in the actual system correspond most directly to the primary interventions? In Table 8, an unescaped "second parsing path" is identified within the Windows environment; however, the current benchmark excludes this environment. Have any POSIX/Bash-based paths been investigated that would allow for the reproduction of these responses using the actual command-processing code? Furthermore, if the target is primarily an SSH wrapper constructed by the model itself, is it possible to distinguish between an erroneous wrapper written by the model and an error introduced by the framework during post-generation processing?
-
What are the statistical criteria for defining a "rank reversal"? In Table 2, the average success rates for GPT-5.6-sol and Gemini-3.5-Flash no longer show a reversal; why, then, is this pair still characterized as exhibiting a clear rank reversal? Could you provide the average success rates, the "all-three-trials-passed" rates, and the confidence intervals for the paired differences across the two paths? Specifically, what stability criteria define the "22 stable pairwise orderings"?
-
Are changes to auxiliary files factored into the final evaluation across all experiments? How can the descriptions in the "Limitations" section and Appendix F.1 be reconciled? Do all experimental campaigns employ the same criteria for directory inspection? Additionally, where are temporary scripts created, and at what point are they cleaned up—could these processes affect the verification results?
What specific types of failures are actually resolved through "model adaptation"? Beyond the average compensation magnitude, could you report the number of tasks that follow an "RR (success) → RN (failure) → NN (success)" pattern, as well as the number of new failures that emerge following a change in the contract? Such task-level transitions would clarify whether the compensation primarily restores functionality lost on the original path, or if it also reflects shifts in the set of successful outcomes resulting from the generation of different commands.
4 comments
W1 / Q1. A direct link to real systems' command processing.
The revision does both remedies you proposed.
| Your remedy | What we did | Where |
|---|---|---|
| Replay saved responses through real command-transformation pipelines | 448 stored replies through 10 open systems' command-handling code at the surveyed commits, in our container with our validators (component replay, zero model calls) | App. A.1.2, Table 9 |
| Frame as a boundary stress test | Abstract, Contribution 1 and Limitations now say so | Abstract, Sections 1, 5 |
Real-code replay results:
| Systems | Result |
|---|---|
| Codex, Goose, mini-swe-agent, Desktop Commander (single shell) | Match the raw path on all 448 |
| The other systems | Differ only by their own tokenization, shell selection, filters and session handling; never by a second parse |
Which real path corresponds most directly? A wrapper the agent writes, e.g. ssh host "..." through Codex. Codex passes the command as one shell -c argument (Table 5), like our raw path; the remote shell parses it again under our nested rules. Such wrappers are common: 27,096 of 53,938 Bash commands in our session logs go through ssh (App. A.1). Agent-written results → App. E.1.5:
| Setting | Result |
|---|---|
| Two SSH-like families, all 8 configurations | 58.3–100% pass |
| 12-family pilot | Flash-Lite 50.0% vs 83.3% raw; Sonnet-4.6 67.4% vs 85.4%; Opus-5 and Opus-4.8 lose nothing |
The real-ssh replay constructs the same wrapper (your W1). That is by design: it tests that the nested path behaves like a real remote command (893 of 896 replies, Tables 7, 28). The case where the model builds the wrapper itself is the agent-written table above.
Model's wrapper or the framework? Rerun the stored command on the raw path with task, environment and validator fixed. A failure there predates any harness transformation; a raw-path pass that fails on the deployed path isolates damage added by the path.
Q4. Task-level transitions.
Compensation mostly restores what the path broke. Trial 0, 448 model–task pairs → Section 4.3, Table 20
| RR → RN → NN | Count |
|---|---|
| Pass → fail → pass (restored) | 164 of 292 |
| Pass → fail → fail | 128 |
| Pass → pass → pass | 113 |
| Pass → pass → fail (new failures) | 10 of 123 (2.2% of 448) |
| Fail → fail → pass | 9 |
| Fail → fail → fail | 24 |
- Explicit hazards recover: json-write (damage −90.6, compensation +50.0), sed-replace (−43.8, +46.9).
- Implicit hazards stay broken: find-glob, grep-count, hostile-filenames.
- Cost on the raw path: across the six compensating configurations, RR 323 → NR 175 passes; 154 RR passes fail as NR and 6 RR failures pass.
Thank you for addressing the statistical and validator concerns. My remaining concern is whether the narrowed contribution provides sufficient insight beyond the known effects of unescaped parsing. To consider raising my score to 3. I need evidence that QuoteBench reveals reproducible failure patterns with practical diagnostic value. The current pilot and aggregate compensation counts do not yet yet establish that connection. I therefore retain 2 score for now.
Thank you for the follow-up. We added a zero-call analysis of the stored replies (App. E.1.7, "Per-task diagnosis"). It shows that QuoteBench gives every model–task pair a diagnosis that repeats across draws, names the fix, and locates failures on wrappers the model writes itself.
1. Beyond unescaped parsing. Unescaped parsing predicts only that a reply breaks on the wrapper, and the wrapper leaves a similar share passing for every configuration (RN 19.6–30.4%, Table 1). The four cells add what that effect cannot: whether this model repairs the reply once told the path.
| Diagnosis | Cells | Fix |
|---|---|---|
| Generation failure | RR fail | Change the model |
| Survives the wrapper | RN pass | None |
| Compensated | RN fail, NN pass | Disclose the path |
| Uncompensated | RN fail, NN fail | Fix the path; a temporary script recovers every reply the wrapper breaks (292 of 292 in trial 0, App. E.1.1) |
2. Reproducible. Diagnosis identical in all three independent draws:
| Configuration | Same diagnosis (of 56) | Fleiss κ | Same compensated/uncompensated split (damaged in all draws) |
|---|---|---|---|
| GPT-5.6-sol | 47 | 0.80 | 30/35 |
| GPT-5.5 | 41 | 0.67 | 26/38 |
| Opus-5 | 44 | 0.73 | 27/35 |
| Gemini-3.1-Pro | 47 | 0.81 | 34/40 |
| Gemini-3.5-Flash | 31 | 0.54 | 21/34 |
| Opus-4.8 | 44 | 0.78 | 29/33 |
| Qwen3.5-27B | 55 | 0.98 | 30/30 |
| Gemini-3.1-Flash-Lite | 53 | 0.94 | 32/32 |
| All | 362/448 (81%) | 229/277 (83%) |
3. Diagnostic value: task-level transfer to agent-written wrappers. The model writes its own ssh-like call, with the same payload and validator (App. E.1.5). This tests each of the 133 failed runs, beyond the four-configuration ranking:
| Configuration | Failed runs | On tasks where the same model failed the disclosed nested path | Expected if random |
|---|---|---|---|
| Opus-5 | 3 | 3 | 0.7 |
| Opus-4.8 | 11 | 11 | 7.1 |
| Gemini-3.1-Flash-Lite | 72 | 72 | 61.5 |
| Sonnet-4.6 (one nested draw) | 47 | 38 | 33.3 |
| All | 133 | 124 | 102.6 (one-sided permutation p < 0.001) |
Practical use. One QuoteBench run tells a team, per operation, whether to change the model, disclose the path, or fix the path. The answer repeats across draws for 81% of model–task pairs, and a model's disclosed-path failures mark where its own wrappers fail. We hope this establishes the connection you asked for.
Thank you for the detailed review. W2/Q2 and W3/Q3 are answered here; W1/Q1 and Q4 in the second part.
W2 / Q2. Statistical criteria for the rank reversal.
We removed "unambiguous" and defined reversal; the robust finding is the separation on the nested path (9 of 15 intervals exclude zero, 0 of 15 on the raw path). A pair reverses when its point estimates take strictly opposite orders on the two paths; ties are not comparable; counts are descriptive. → Section 4.2, App. E.1.7
GPT-5.6-sol vs Gemini-3.5-Flash, separately for each reading you asked for:
W3 / Q3. One definition of success; temporary scripts; consistency across campaigns.
Section 3.1 now states one rule, and the Limitations sentence matches it. → Section 3.1
.qb-transport-script.sh, written in the working directory, deleted before validation, never counted as a new file; script replays match the raw path on all 448 public and 126 private pairs (App. E.1.1)Which campaigns apply the new-file check → App. F.1
sshreplays