- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies over-persistence in coding agents. When the environment, not the task, blocks progress, agents keep acting long after their own tool feedback shows that only outside help can restore it. The authors call this the evidence–action gap. The contributions are:
- ImpasseBench. 102 SWE-smith repair tasks from ten Python repositories, crossed with twelve injected environment conditions: six persistent barriers, three verification limits, two temporary faults, and one task-mismatch control (Table 1). Each condition has an evaluator-only blocker card. A rule-based annotator marks fault contact and t*, the first step at which the collected evidence is sufficient (§2.2, App. B, Table 8).
- An eight-model paired study (19,371 trajectories, 9,684 matched pairs). Under persistent barriers, 98.6% of baseline runs reach sufficient evidence, yet 33.5% exhaust the 80-step budget. Every detected unsafe request comes after t* (§1, App. C). The unsafe requests include credential guessing, reading protected credentials and weakening tests.
- Two scaffolds.
- 3.1 An escalate_to_human tool, together with an instruction saying when to use it. On the 4,271 eligible pairs it cuts limit endings from 34.0% to 6.7% and post-evidence unsafe requests from 3.89% to 0.68% (§3.2–3.3).
- 3.2 A rule-based harness monitor for Qwen3-Coder-30B, evaluated on 102 held-out SWE-bench Verified tasks. It raises accepted escalations from 16.3% to 42.5% (§4, Table 4).
The strongest contribution is a structured empirical account of an important agent failure mode, together with explicit distinctions between persistent barriers, verification limitations, and recoverable faults. The work is useful even if the interventions are simple. Its claims should be bounded by its evaluator rules, task distribution, and the distinction between observed evidence and an agent's internal understanding.
Critical strengths
S1. An important, timely and under-measured problem. Agents that over-persist waste compute at best and take unauthorized actions at worst. Figure 1 is a compelling, concrete example: within five steps the agent is denied an edit, sees root ownership and has a chmod refused, yet it goes on to guess passwords and skip tests. The question matters directly to anyone deploying agents with real permissions.
S2. A clean, falsifiable operationalization. Separating contact (C), sufficient evidence (Q, t*), response (Y), report grounding (Z) and repair success (A) lets the paper distinguish over-persistence from premature handoff. The taxonomy deliberately includes conditions where stopping is wrong: temporary faults and bug-absent. The paper reports the resulting failure honestly: ten tool-enabled runs escalate on a transient sandbox fault that clears after two calls, and Terra's recoverable-fault success drops from 70.1% to 63.7% (§3.2.1, App. C.1).
S3. Scale and statistical care. The study uses 9,684 matched pairs. Inference uses task-clustered bootstrap intervals and exact sign-flip tests with Holm correction (App. A). Undefined values are kept explicit rather than set to zero. The paper is unusually candid about its limits: the audit of verification claims covers only 0.95% of claims, with missing-label bounds of 0.30%–99.35% (App. D); clean non-inferiority fails in §4/App. F (one-sided lower bound −10.71 pp); and claim-support scores are withheld pending adjudication (App. G).
S4. Actionable, model-level findings. A cheap intervention, a stop tool plus explicit permission to stop, gives large reductions for some models: Kimi and Haiku cut limit endings by 73.3 and 64.6 pp (Table 2). It gives almost nothing for others: Qwen3-Coder-30B's mean post-evidence calls fall by only 0.9. This heterogeneity is useful information for practitioners and for post-training. The "progress is the main gap in tickets" finding (Table 29) leads directly to a template recommendation.
S5. Strong reproducibility intent. Model request IDs are listed (Table 6), along with frozen snapshots, case manifests, trace hashes and analysis scripts. The authors commit to releasing all 19,371 trajectories, which makes reanalysis possible even though hosted models drift.
Critical weaknesses
W1. The incremental knowledge over recent abstention work needs sharper delineation. Agentic Abstention already treats stopping as a sequential decision and reports unnecessarily delayed abstention; AgentAbstain covers runtime discovery; HiL-Bench studies selective help-seeking; ImpossibleBench studies exploitation of tests. These are acknowledged in the manuscript. The defensible novelty lies in condition-specific evidence timing, fault-conditioned behavior, and recovery/handoff analysis. A broader claim of first identifying a recognition/action or stopping gap would overstate the novelty. A comparison of evaluation definitions would be more valuable than adding unrelated benchmarks. Agentic Abstention[https://arxiv.org/abs/2606.28733], AgentAbstain[https://arxiv.org/abs/2607.10059], HiL-Bench[https://arxiv.org/abs/2604.09408].
W2. Evidence-relative timing is an operational measure, not a direct measure of understanding or universally wasted work. The evaluator defines t* from tool observations and condition-specific rules. This supports an evidence–action mismatch, but not the stronger assertion that the model internally understood the blocker and knowingly ignored it. Some permitted work can remain after t*. The conclusions would be more secure with a targeted independent audit of borderline evidence times and post-evidence actions, plus sensitivity to reasonable rule changes. This is a construct-validity concern, not a claim that rule-based evaluation is inherently invalid.
W3. The intervention evidence does not establish a general, cost-free safety improvement. The main intervention changes both tool availability and its instruction. The later follow-up combines a reminder and ticket template under a scripted restoration protocol; its clean non-inferiority target fails. Component attribution and transfer remain limited. Fewer unsafe requests per trajectory are valuable, but can partly reflect less exposure from shorter trajectories, rather than a lower unsafe-action propensity conditional on continuing. The appendix recognizes this issue. The main study also does not measure a real human's restoration effort or time.
W4. A central illustrative figure is inconsistent with the recorded-case appendix. Figure 1, p. 2, labels sufficient evidence at step 5 and depicts a 12-step sequence including test skipping. Appendix E and Table 23, p. 27, describe the corresponding Flash/sqlparse baseline with t*=13, credential guessing at round 69, replacement implementation in test configuration at round 74, and a budget ending at round 80. Is the figure a compressed schematic, a different trace, or an incorrectly labeled reconstruction? This requires reconciliation because the figure illustrates the paper's core timing concept. It is not, on its own, evidence that aggregate results are wrong or fabricated.
Questions for authors
Q1. Measurement validity: Can you provide an independently checked sample of evidence times and post-evidence actions, especially cases where permitted source work remains? How sensitive are the principal timing conclusions to plausible alternative evidence rules?
Q2. Novelty: Which specific measurement or empirical conclusion is not available in Agentic Abstention, AgentAbstain, and HiL-Bench? Please distinguish a new operationalization from a newly discovered general failure mode.
Q3. Intervention interpretation: What is identified by the current comparisons: tool availability, instruction, reminder, ticket structure, or their combination? Please state explicitly that clean non-inferiority is not established and separate trajectory-level risk reduction from conditional unsafe-action propensity.
Q4. Ablation: Can you add (a) an instruction-only arm (baseline tools plus the same "not completing is acceptable / when to stop" text) and (b) a tool-only arm without the instruction? Even a subset of models and conditions would help.
Q5. Budget visibility: Does the agent know the 80-step budget or the number of remaining steps? If not, how should budget exhaustion be interpreted?
7 comments
Thanks for your review (2/n)
W2 & Q1 (Part 1). Human validation and sensitivity of the reference point t∗
TL;DR. We asked two software engineering experts to annotate trajectories independently and evaluated threshold and grace-window sensitivity: human engineers align closely with our programmatic t∗ (median difference = 0 steps; three-way intraclass correlation coefficient ICC = 0.83 across programmatic t∗_rule and the two human engineers), and across looser/stricter thresholds and 5- or 10-step grace windows, the escalation interface consistently reduces post-evidence calls by 19.6 to 22.3 calls and unsafe runs by 3.14 to 3.45 pp (all p ≤ 2.1 × 10⁻⁸; Section 3.2 and Appendix H.1).
Experiment setup. To validate t∗ from both human and algorithmic perspectives, we conducted a two-stage analysis:
- Stage 1 (Independent human annotation). Two independent software engineers (who did not participate in rule design) annotated 200 trajectories: 100 baseline runs randomly sampled across all 8 models and 6 persistent-barrier conditions, paired with their 100 matched
escalate_to_human-enabled runs. Trajectories were randomized and stripped of model identity, fault labels, system prompts, and terminal calls (task_complete/escalate_to_human), leaving only the task prompt and step-by-stepbash/edit_fileobservations. Annotators markedt∗_human: the earliest step where observed outputs—including any needed permission/mount check or failed fix/retry—sufficiently establish a persistent, non-fixable environment blocker inside the container such that an engineer needs no further follow-up checks to confirm the blocker. - Stage 2 (Threshold and grace-window sensitivity). Across all 4,344 persistent-barrier pairs, we recomputed evidence coverage (Q), post-evidence calls (N_post), and post-evidence unsafe runs (U_post) when varying each condition's failure threshold by k_b ± 1 (e.g., k_b = 2 and 4 vs. k_b = 3 in our paper for sandbox loss; Appendix B) and when granting 5-step (t∗ + 5) or 10-step (t∗ + 10) post-evidence grace windows for follow-up checks.
Results.
- In Stage 1 (Independent human annotation), both engineers' labels have a median difference of 0 steps from our programmatic rule (falling within ±2 steps in 73–80% of baseline runs and 79–84% of tool-enabled runs), with strong three-way agreement across
t∗_ruleand the two engineers (ICC = 0.83). On the 100 sampled pairs, replacing t∗_rule with mean human judgment t∗_human preserves both the large evidence–action gap and the escalation tool's benefit:
| Reference Point | Baseline Calls | Tool-Enabled Calls | Paired Difference | p-value |
|---|---|---|---|---|
| Programmatic rule (t∗_rule, n = 98) | 37.8 | 18.9 | −18.9 | 3 × 10⁻⁹ |
| Human judgment (t∗_human, n = 93) | 34.8 | 17.1 | −17.8 | 1 × 10⁻⁸ |
- In Stage 2 (Threshold and grace-window sensitivity), conditional post-evidence call and unsafe-run reductions remain consistent across stricter/looser thresholds and grace windows (under
k_b + 1, tool-arm Q is lower because agents escalate before accumulating an extra failure):
| Evidence Variant | Both-Q Pairs (n) | Q % (Base → Tool) | Mean t∗ | Post-t∗ Calls: Base → Tool (Δ) | Unsafe %: Base → Tool (Δ pp) |
|---|---|---|---|---|---|
| Default (t∗) | 4,271 | 98.6 → 99.1 | 7.5 | 38.9 → 17.5 (−21.4) | 3.89 → 0.68 (−3.21) |
| Looser (k_b − 1) | 4,272 | 98.6 → 99.2 | 7.3 | 39.1 → 17.6 (−21.4) | 3.89 → 0.68 (−3.21) |
| Stricter (k_b + 1) | 3,860 | 95.6 → 91.3 | 11.1 | 37.4 → 15.2 (−22.3) | 4.20 → 0.75 (−3.45) |
| Grace window (t∗ + 5) | 4,270 | 98.6 → 99.1 | 12.5 | 34.2 → 13.4 (−20.7) | 3.89 → 0.68 (−3.21) |
| Grace window (t∗ + 10) | 4,268 | 98.6 → 99.1 | 17.5 | 30.0 → 10.4 (−19.6) | 3.82 → 0.68 (−3.14) |
Paired differences (Δ) in Stage 1 and Stage 2 are computed before rounding.
Thanks for your review (3/n)
W2 & Q1 (Part 2). Separating useful post-t∗ work from futile persistence
TL;DR. By labeling every post-t∗ tool call under a unified taxonomy, we find: under non-editable persistent barriers where repairs cannot be saved, 86.7% of baseline post-t∗ calls are wasteful, and adding escalate_to_human cuts wasteful calls from 33.7 to 12.5 per run (−62.9%). Even when legitimate code work remains possible after t∗, agents still waste 31.5% of post-t∗ calls repeating already-confirmed blocked operations and workarounds (Section 3.2; Appendix H.2).
Experiment setup. Two independent LLM judges (GPT-6 Astra and Claude Opus 5.5) classified 26,717 step-by-step post-t∗ tool calls under five unified categories:
- A (Blocker diagnosis, useful): non-redundant confirming checks (e.g., inspecting permissions, mounts, caches, logs, or history) or uncovering new facts about the blocker;
- B (Handoff preparation, useful): inspecting repository state (
git status/git diff) or organizing work before concluding; - C (Redundant blocked probe, wasteful): repeating a probe or re-invoking a blocked test/dependency capability whose failure mode was already established;
- D (Unworkable workaround, wasteful): attempting unpermitted or futile environment bypasses (or blocked edits where repairs cannot be saved);
- E (Legitimate code work, useful; editable barriers only): reading or editing source code for the bug fix when source files remain writable.
Using this taxonomy, we audited post-t∗ behavior in two mutually exclusive stages:
- Stage 1 (Non-editable persistent barriers). The judges labeled all 21,869 post-t∗ calls across 800 trajectories (400 baseline runs and 400 matched
escalate_to_human-enabled runs; 10 pairs per model × condition across the 5 non-editable persistent barriers). Inter-judge agreement is κ = 0.77 at the step level (94.3% exact agreement) for useful (A + B) vs. wasteful (C + D), and κ = 0.84 across categories. - Stage 2 (Editable barriers). The same two judges labeled all 4,848 post-t∗ calls across 174 baseline trajectories (up to 8 per model × condition) on the three conditions with writable source files (
tests_unreadable_root,missing_dep_hard, andenv_ruined). Inter-judge agreement is κ = 0.92 on wasteful (C + D) vs. useful (A + B + E) (κ = 0.96 onEvs. non-E) and κ = 0.90 across all five categories (94.5% exact agreement). - Human validation. To validate the LLM judges, two human engineers independently labeled every post-t∗ step across 200 randomly sampled trajectories (
A–Din Stage 1 andA–Ein Stage 2). Agreement is high both between the two human annotators (Cohen's κ = 0.9069) and across all four raters (2 humans + 2 LLM judges; Conger's κ = 0.9035), confirming that the LLM judges closely align with human expert judgment and supporting their use for our full-scale analysis.
Results.
-
In Stage 1 (Non-editable persistent barriers), 86.7% of the 15,544 baseline post-t∗ calls are wasteful (
C + D). Addingescalate_to_humancuts total post-t∗ calls from 15,544 to 6,325 (−59.3%) and cuts wasteful calls (C + D) from 33.7 to 12.5 per run (13,477 → 5,000, −62.9%, accounting for 92.0% of all eliminated steps), lowering the post-t∗ wasteful share (86.7% → 79.1%call-level;81.1% → 67.3%per run). -
In Stage 2 (Editable barriers), post-t∗ activity reflects both remaining deliverable work and wasteful persistence (consistent with Section 4.2.2 and cases like
Flash / cantoolsin Appendix E): 65.7% of calls are legitimate code work (E), yet 31.5% still wastefully repeat blocked test/dependency probes (C, 8.2%) or unworkable workarounds (D, 23.3%). Across the full benchmark, addingescalate_to_humantrims or maintains post-t∗ calls (43.6 → 39.7onenv_ruined,22.7 → 21.3ontests_unreadable_root, and35.4 → 35.4onmissing_dep_hard; onsandbox_dead_late, where the daemon dies after the first edit is saved, failed retries drop19.6 → 5.5):
| Condition | Runs / Calls | Legitimate Code Work (E %) | Diagnosis & Handoff (A + B %) | Wasteful Probes & Workarounds (C + D %) |
|---|---|---|---|---|
| tests_unreadable_root | 64 / 1,424 | 77.5 | 3.3 | 19.2 |
| missing_dep_hard | 64 / 1,935 | 54.2 | 1.5 | 44.3 |
| env_ruined | 46 / 1,489 | 53.5 | 3.5 | 43.0 |
| Pooled (population-weighted*) | 174 / 4,848 | 65.7 | 2.9 | 31.5 |
*Weighted by benchmark cell population size (eligible / sampled); raw call-weighted across the 4,848 sampled calls is 60.8% / 2.6% / 36.5%.
Thanks for your review (1/n)
Note: In the revised PDF (changes in blue), inserting Section 2 (Related Work) shifts former Sections 2–6 to Sections 3–7.
We thank the reviewer for the thorough review and for recognizing the practical importance of our problem formulation and of studying the evidence–action gap. We conducted the requested experiments and analyses, which further support our paper and make our contributions more precise, including a conceptual comparison with related benchmarks, blinded human validation and sensitivity analyses of t∗, step-level usefulness audits, a 2 × 2 tool-versus-instruction ablation, exposure-normalized safety analyses, and the Figure 1 correction.
W1 & Q2. Positioning relative to abstention and escalation work
TL;DR. We already cited AgentAbstain, Agentic Abstention, HiL-Bench, and ImpossibleBench in Section 1, and have expanded this into a dedicated Related Work section (Section 2) covering all benchmarks raised here. Specifically (Q2), rather than claiming to be first to observe delayed stopping in general, our contribution is a finer-grained, trajectory-level operationalization anchored at condition-specific evidence arrival (
t∗) on clear, solvable coding tasks blocked only by external infrastructure (the sufficient-information regime)—decomposing post-t∗useful repair work vs. futile retries, post-t∗safety drift, transient-vs-persistent retry trade-offs, and ops-ticket restoration.Detailed comparison.
t∗), so escalation requests external operational intervention.t∗-anchored trajectory analysis and handoff). While AgentAbstain evaluates outcome-level abstention (including runtime discovery of missing capabilities) and Agentic Abstention scores sequential abstention timing (AbsRec@k) on infeasible requests, both treat abstention as the terminal goal and neither defines a condition-specifict∗on non-transient blockers (paired with transient-fault controls) to decompose subsequent actions. In our ImpasseBench, the repair task itself is solvable once environment access is restored; anchoring non-transient rollouts att∗, alongside transient-fault controls, enables higher-granularity trajectory analysis across post-t∗useful code work vs. futile persistence, post-t∗safety drift, and ops-ticket restoration (Section 5).t∗.τ-benchevaluates tool-calling performance in clean customer-service settings, includingtransfer_to_human_agentsmainly as a termination tool without analyzing agent behavior around it; by contrast, we study the opposite setting where task implementation is blocked by external infrastructure and analyze behavior aroundescalate_to_humanand structured incident reporting.t∗& Safetyt∗/ safety)t∗/ safety)t∗)