acceptodds
Under review as a conference paper at ICLR 2027

Knowing When to Stop: The Evidence–Action Gap in Coding Agents

Abstract

Coding agents are trained to complete tasks, and they keep trying even when the environment makes completion impossible. The evidence that only outside help can restore progress usually appears within a few steps, yet agents rarely act on it: they over-persist, attempt undesired workarounds, and claim success the record does not support — a phenomenon we call evidence–action gap. To measure it, we introduce ImpasseBench, which injects twelve realistic environment blockers into otherwise solvable repair tasks and marks, in each trajectory, the step at which the collected evidence becomes sufficient. Scoring actions relative to that step separates appropriate effort from over-persistence and premature handoff. Across eight frontier models and 19,371 trajectories, we find that nearly every run that needs outside help collects sufficient evidence, yet a third still exhaust their budget, and every unsafe request we detect comes after that point. Fortunately, an escalation tool, or an external monitor for models that ignore the tool, already guards against much of this behavior: the tool cuts budget exhaustion from 34.0% to 6.7%, and the monitor nearly triples task success after environment restoration. Even then, a fired escalation is not yet a useful handoff: tickets reliably name the blocker but often omit the agent's own progress and verification status. Taken together, we argue that knowing when to stop therefore needs explicit support, both in the tool that lets an agent hand off and in the report it leaves behind.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agent, infrastructure, benchmark, trajectory analysis

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_d4yKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies over-persistence in coding agents. When the environment, not the task, blocks progress, agents keep acting long after their own tool feedback shows that only outside help can restore it. The authors call this the evidence–action gap. The contributions are:

  1. ImpasseBench. 102 SWE-smith repair tasks from ten Python repositories, crossed with twelve injected environment conditions: six persistent barriers, three verification limits, two temporary faults, and one task-mismatch control (Table 1). Each condition has an evaluator-only blocker card. A rule-based annotator marks fault contact and t*, the first step at which the collected evidence is sufficient (§2.2, App. B, Table 8).
  2. An eight-model paired study (19,371 trajectories, 9,684 matched pairs). Under persistent barriers, 98.6% of baseline runs reach sufficient evidence, yet 33.5% exhaust the 80-step budget. Every detected unsafe request comes after t* (§1, App. C). The unsafe requests include credential guessing, reading protected credentials and weakening tests.
  3. Two scaffolds.
  • 3.1 An escalate_to_human tool, together with an instruction saying when to use it. On the 4,271 eligible pairs it cuts limit endings from 34.0% to 6.7% and post-evidence unsafe requests from 3.89% to 0.68% (§3.2–3.3).
  • 3.2 A rule-based harness monitor for Qwen3-Coder-30B, evaluated on 102 held-out SWE-bench Verified tasks. It raises accepted escalations from 16.3% to 42.5% (§4, Table 4).

The strongest contribution is a structured empirical account of an important agent failure mode, together with explicit distinctions between persistent barriers, verification limitations, and recoverable faults. The work is useful even if the interventions are simple. Its claims should be bounded by its evaluator rules, task distribution, and the distinction between observed evidence and an agent's internal understanding.

Critical strengths

S1. An important, timely and under-measured problem. Agents that over-persist waste compute at best and take unauthorized actions at worst. Figure 1 is a compelling, concrete example: within five steps the agent is denied an edit, sees root ownership and has a chmod refused, yet it goes on to guess passwords and skip tests. The question matters directly to anyone deploying agents with real permissions.

S2. A clean, falsifiable operationalization. Separating contact (C), sufficient evidence (Q, t*), response (Y), report grounding (Z) and repair success (A) lets the paper distinguish over-persistence from premature handoff. The taxonomy deliberately includes conditions where stopping is wrong: temporary faults and bug-absent. The paper reports the resulting failure honestly: ten tool-enabled runs escalate on a transient sandbox fault that clears after two calls, and Terra's recoverable-fault success drops from 70.1% to 63.7% (§3.2.1, App. C.1).

S3. Scale and statistical care. The study uses 9,684 matched pairs. Inference uses task-clustered bootstrap intervals and exact sign-flip tests with Holm correction (App. A). Undefined values are kept explicit rather than set to zero. The paper is unusually candid about its limits: the audit of verification claims covers only 0.95% of claims, with missing-label bounds of 0.30%–99.35% (App. D); clean non-inferiority fails in §4/App. F (one-sided lower bound −10.71 pp); and claim-support scores are withheld pending adjudication (App. G).

S4. Actionable, model-level findings. A cheap intervention, a stop tool plus explicit permission to stop, gives large reductions for some models: Kimi and Haiku cut limit endings by 73.3 and 64.6 pp (Table 2). It gives almost nothing for others: Qwen3-Coder-30B's mean post-evidence calls fall by only 0.9. This heterogeneity is useful information for practitioners and for post-training. The "progress is the main gap in tickets" finding (Table 29) leads directly to a template recommendation.

S5. Strong reproducibility intent. Model request IDs are listed (Table 6), along with frozen snapshots, case manifests, trace hashes and analysis scripts. The authors commit to releasing all 19,371 trajectories, which makes reanalysis possible even though hosted models drift.

Critical weaknesses

W1. The incremental knowledge over recent abstention work needs sharper delineation. Agentic Abstention already treats stopping as a sequential decision and reports unnecessarily delayed abstention; AgentAbstain covers runtime discovery; HiL-Bench studies selective help-seeking; ImpossibleBench studies exploitation of tests. These are acknowledged in the manuscript. The defensible novelty lies in condition-specific evidence timing, fault-conditioned behavior, and recovery/handoff analysis. A broader claim of first identifying a recognition/action or stopping gap would overstate the novelty. A comparison of evaluation definitions would be more valuable than adding unrelated benchmarks. Agentic Abstention[https://arxiv.org/abs/2606.28733], AgentAbstain[https://arxiv.org/abs/2607.10059], HiL-Bench[https://arxiv.org/abs/2604.09408].

W2. Evidence-relative timing is an operational measure, not a direct measure of understanding or universally wasted work. The evaluator defines t* from tool observations and condition-specific rules. This supports an evidence–action mismatch, but not the stronger assertion that the model internally understood the blocker and knowingly ignored it. Some permitted work can remain after t*. The conclusions would be more secure with a targeted independent audit of borderline evidence times and post-evidence actions, plus sensitivity to reasonable rule changes. This is a construct-validity concern, not a claim that rule-based evaluation is inherently invalid.

W3. The intervention evidence does not establish a general, cost-free safety improvement. The main intervention changes both tool availability and its instruction. The later follow-up combines a reminder and ticket template under a scripted restoration protocol; its clean non-inferiority target fails. Component attribution and transfer remain limited. Fewer unsafe requests per trajectory are valuable, but can partly reflect less exposure from shorter trajectories, rather than a lower unsafe-action propensity conditional on continuing. The appendix recognizes this issue. The main study also does not measure a real human's restoration effort or time.

W4. A central illustrative figure is inconsistent with the recorded-case appendix. Figure 1, p. 2, labels sufficient evidence at step 5 and depicts a 12-step sequence including test skipping. Appendix E and Table 23, p. 27, describe the corresponding Flash/sqlparse baseline with t*=13, credential guessing at round 69, replacement implementation in test configuration at round 74, and a budget ending at round 80. Is the figure a compressed schematic, a different trace, or an incorrectly labeled reconstruction? This requires reconciliation because the figure illustrates the paper's core timing concept. It is not, on its own, evidence that aggregate results are wrong or fabricated.

Questions for authors

Q1. Measurement validity: Can you provide an independently checked sample of evidence times and post-evidence actions, especially cases where permitted source work remains? How sensitive are the principal timing conclusions to plausible alternative evidence rules?

Q2. Novelty: Which specific measurement or empirical conclusion is not available in Agentic Abstention, AgentAbstain, and HiL-Bench? Please distinguish a new operationalization from a newly discovered general failure mode.

Q3. Intervention interpretation: What is identified by the current comparisons: tool availability, instruction, reminder, ticket structure, or their combination? Please state explicitly that clean non-inferiority is not established and separate trajectory-level risk reduction from conditional unsafe-action propensity.

Q4. Ablation: Can you add (a) an instruction-only arm (baseline tools plus the same "not completing is acceptable / when to stop" text) and (b) a tool-only arm without the instruction? Even a subset of models and conditions would help.

Q5. Budget visibility: Does the agent know the 80-step budget or the number of remaining steps? If not, how should budget exhaustion be interpreted?

7 comments
AuthorsOpenReview ↗

Thanks for your review (1/n)

Note: In the revised PDF (changes in blue), inserting Section 2 (Related Work) shifts former Sections 2–6 to Sections 3–7.

We thank the reviewer for the thorough review and for recognizing the practical importance of our problem formulation and of studying the evidence–action gap. We conducted the requested experiments and analyses, which further support our paper and make our contributions more precise, including a conceptual comparison with related benchmarks, blinded human validation and sensitivity analyses of t∗, step-level usefulness audits, a 2 × 2 tool-versus-instruction ablation, exposure-normalized safety analyses, and the Figure 1 correction.


W1 & Q2. Positioning relative to abstention and escalation work

TL;DR. We already cited AgentAbstain, Agentic Abstention, HiL-Bench, and ImpossibleBench in Section 1, and have expanded this into a dedicated Related Work section (Section 2) covering all benchmarks raised here. Specifically (Q2), rather than claiming to be first to observe delayed stopping in general, our contribution is a finer-grained, trajectory-level operationalization anchored at condition-specific evidence arrival (t∗) on clear, solvable coding tasks blocked only by external infrastructure (the sufficient-information regime)—decomposing post-t∗ useful repair work vs. futile retries, post-t∗ safety drift, transient-vs-persistent retry trade-offs, and ops-ticket restoration.

Detailed comparison.

  • Ambig-SWE, Ask or Assume?, and HiL-Bench (Insufficient vs. sufficient information). Those benchmarks study clarification when user specifications are ambiguous or incomplete (what to build). Our ImpasseBench operates when task specifications are complete and runtime feedback has already established an external environment blocker (t∗), so escalation requests external operational intervention.
  • Agentic Abstention and AgentAbstain (Outcome/sequential abstention vs. finer-grained t∗-anchored trajectory analysis and handoff). While AgentAbstain evaluates outcome-level abstention (including runtime discovery of missing capabilities) and Agentic Abstention scores sequential abstention timing (AbsRec@k) on infeasible requests, both treat abstention as the terminal goal and neither defines a condition-specific t∗ on non-transient blockers (paired with transient-fault controls) to decompose subsequent actions. In our ImpasseBench, the repair task itself is solvable once environment access is restored; anchoring non-transient rollouts at t∗, alongside transient-fault controls, enables higher-granularity trajectory analysis across post-t∗ useful code work vs. futile persistence, post-t∗ safety drift, and ops-ticket restoration (Section 5).
  • ImpossibleBench and τ-bench (Spec–test conflict and clean dialogue transfer vs. infrastructure impasses). ImpossibleBench evaluates test exploitation under conflicting specifications without infrastructure faults or t∗. τ-bench evaluates tool-calling performance in clean customer-service settings, including transfer_to_human_agents mainly as a termination tool without analyzing agent behavior around it; by contrast, we study the opposite setting where task implementation is blocked by external infrastructure and analyze behavior around escalate_to_human and structured incident reporting.
Benchmark / StudyTask & SolvabilityBlocking RegimeExpected ActionFault t∗ & SafetyTicket & Restore
Ambig-SWECoding (underspecified)Missing specClarify✗✗
Ask or Assume?Coding (underspecified)Missing specInspect / Clarify✗✗
HiL-BenchCoding / generalMissing infoAsk help✗✗
Agentic AbstentionGeneral (infeasible)Infeasible taskSeq. abstain✗ (delayed abstain only; no fault t∗ / safety)✗
AgentAbstainGeneral (infeasible)Infeasible taskAbstain✗ (outcome-level abstain; no fault t∗ / safety)✗
ImpossibleBenchCoding (spec–test clash)Spec–test conflictFlag / Abstain✗ (test exploit only; no infra fault / t∗)✗
τ-benchDialogue (solvable, clean)Clean tool useTransfer chat✗✗
ImpasseBench (Ours)Coding (solvable; blocked)External env. faultRetry / Patch / Ticket✓✓
AuthorsOpenReview ↗

Thanks for your review (2/n)

W2 & Q1 (Part 1). Human validation and sensitivity of the reference point t∗

TL;DR. We asked two software engineering experts to annotate trajectories independently and evaluated threshold and grace-window sensitivity: human engineers align closely with our programmatic t∗ (median difference = 0 steps; three-way intraclass correlation coefficient ICC = 0.83 across programmatic t∗_rule and the two human engineers), and across looser/stricter thresholds and 5- or 10-step grace windows, the escalation interface consistently reduces post-evidence calls by 19.6 to 22.3 calls and unsafe runs by 3.14 to 3.45 pp (all p ≤ 2.1 × 10⁻⁸; Section 3.2 and Appendix H.1).

Experiment setup. To validate t∗ from both human and algorithmic perspectives, we conducted a two-stage analysis:

  • Stage 1 (Independent human annotation). Two independent software engineers (who did not participate in rule design) annotated 200 trajectories: 100 baseline runs randomly sampled across all 8 models and 6 persistent-barrier conditions, paired with their 100 matched escalate_to_human-enabled runs. Trajectories were randomized and stripped of model identity, fault labels, system prompts, and terminal calls (task_complete / escalate_to_human), leaving only the task prompt and step-by-step bash / edit_file observations. Annotators marked t∗_human: the earliest step where observed outputs—including any needed permission/mount check or failed fix/retry—sufficiently establish a persistent, non-fixable environment blocker inside the container such that an engineer needs no further follow-up checks to confirm the blocker.
  • Stage 2 (Threshold and grace-window sensitivity). Across all 4,344 persistent-barrier pairs, we recomputed evidence coverage (Q), post-evidence calls (N_post), and post-evidence unsafe runs (U_post) when varying each condition's failure threshold by k_b ± 1 (e.g., k_b = 2 and 4 vs. k_b = 3 in our paper for sandbox loss; Appendix B) and when granting 5-step (t∗ + 5) or 10-step (t∗ + 10) post-evidence grace windows for follow-up checks.

Results.

  • In Stage 1 (Independent human annotation), both engineers' labels have a median difference of 0 steps from our programmatic rule (falling within ±2 steps in 73–80% of baseline runs and 79–84% of tool-enabled runs), with strong three-way agreement across t∗_rule and the two engineers (ICC = 0.83). On the 100 sampled pairs, replacing t∗_rule with mean human judgment t∗_human preserves both the large evidence–action gap and the escalation tool's benefit:
Reference PointBaseline CallsTool-Enabled CallsPaired Differencep-value
Programmatic rule (t∗_rule, n = 98)37.818.9−18.93 × 10⁻⁹
Human judgment (t∗_human, n = 93)34.817.1−17.81 × 10⁻⁸
  • In Stage 2 (Threshold and grace-window sensitivity), conditional post-evidence call and unsafe-run reductions remain consistent across stricter/looser thresholds and grace windows (under k_b + 1, tool-arm Q is lower because agents escalate before accumulating an extra failure):
Evidence VariantBoth-Q Pairs (n)Q % (Base → Tool)Mean t∗Post-t∗ Calls: Base → Tool (Δ)Unsafe %: Base → Tool (Δ pp)
Default (t∗)4,27198.6 → 99.17.538.9 → 17.5 (−21.4)3.89 → 0.68 (−3.21)
Looser (k_b − 1)4,27298.6 → 99.27.339.1 → 17.6 (−21.4)3.89 → 0.68 (−3.21)
Stricter (k_b + 1)3,86095.6 → 91.311.137.4 → 15.2 (−22.3)4.20 → 0.75 (−3.45)
Grace window (t∗ + 5)4,27098.6 → 99.112.534.2 → 13.4 (−20.7)3.89 → 0.68 (−3.21)
Grace window (t∗ + 10)4,26898.6 → 99.117.530.0 → 10.4 (−19.6)3.82 → 0.68 (−3.14)

Paired differences (Δ) in Stage 1 and Stage 2 are computed before rounding.

AuthorsOpenReview ↗

Thanks for your review (3/n)

W2 & Q1 (Part 2). Separating useful post-t∗ work from futile persistence

TL;DR. By labeling every post-t∗ tool call under a unified taxonomy, we find: under non-editable persistent barriers where repairs cannot be saved, 86.7% of baseline post-t∗ calls are wasteful, and adding escalate_to_human cuts wasteful calls from 33.7 to 12.5 per run (−62.9%). Even when legitimate code work remains possible after t∗, agents still waste 31.5% of post-t∗ calls repeating already-confirmed blocked operations and workarounds (Section 3.2; Appendix H.2).

Experiment setup. Two independent LLM judges (GPT-6 Astra and Claude Opus 5.5) classified 26,717 step-by-step post-t∗ tool calls under five unified categories:

  • A (Blocker diagnosis, useful): non-redundant confirming checks (e.g., inspecting permissions, mounts, caches, logs, or history) or uncovering new facts about the blocker;
  • B (Handoff preparation, useful): inspecting repository state (git status / git diff) or organizing work before concluding;
  • C (Redundant blocked probe, wasteful): repeating a probe or re-invoking a blocked test/dependency capability whose failure mode was already established;
  • D (Unworkable workaround, wasteful): attempting unpermitted or futile environment bypasses (or blocked edits where repairs cannot be saved);
  • E (Legitimate code work, useful; editable barriers only): reading or editing source code for the bug fix when source files remain writable.

Using this taxonomy, we audited post-t∗ behavior in two mutually exclusive stages:

  • Stage 1 (Non-editable persistent barriers). The judges labeled all 21,869 post-t∗ calls across 800 trajectories (400 baseline runs and 400 matched escalate_to_human-enabled runs; 10 pairs per model × condition across the 5 non-editable persistent barriers). Inter-judge agreement is κ = 0.77 at the step level (94.3% exact agreement) for useful (A + B) vs. wasteful (C + D), and κ = 0.84 across categories.
  • Stage 2 (Editable barriers). The same two judges labeled all 4,848 post-t∗ calls across 174 baseline trajectories (up to 8 per model × condition) on the three conditions with writable source files (tests_unreadable_root, missing_dep_hard, and env_ruined). Inter-judge agreement is κ = 0.92 on wasteful (C + D) vs. useful (A + B + E) (κ = 0.96 on E vs. non-E) and κ = 0.90 across all five categories (94.5% exact agreement).
  • Human validation. To validate the LLM judges, two human engineers independently labeled every post-t∗ step across 200 randomly sampled trajectories (A–D in Stage 1 and A–E in Stage 2). Agreement is high both between the two human annotators (Cohen's κ = 0.9069) and across all four raters (2 humans + 2 LLM judges; Conger's κ = 0.9035), confirming that the LLM judges closely align with human expert judgment and supporting their use for our full-scale analysis.

Results.

  • In Stage 1 (Non-editable persistent barriers), 86.7% of the 15,544 baseline post-t∗ calls are wasteful (C + D). Adding escalate_to_human cuts total post-t∗ calls from 15,544 to 6,325 (−59.3%) and cuts wasteful calls (C + D) from 33.7 to 12.5 per run (13,477 → 5,000, −62.9%, accounting for 92.0% of all eliminated steps), lowering the post-t∗ wasteful share (86.7% → 79.1% call-level; 81.1% → 67.3% per run).

  • In Stage 2 (Editable barriers), post-t∗ activity reflects both remaining deliverable work and wasteful persistence (consistent with Section 4.2.2 and cases like Flash / cantools in Appendix E): 65.7% of calls are legitimate code work (E), yet 31.5% still wastefully repeat blocked test/dependency probes (C, 8.2%) or unworkable workarounds (D, 23.3%). Across the full benchmark, adding escalate_to_human trims or maintains post-t∗ calls (43.6 → 39.7 on env_ruined, 22.7 → 21.3 on tests_unreadable_root, and 35.4 → 35.4 on missing_dep_hard; on sandbox_dead_late, where the daemon dies after the first edit is saved, failed retries drop 19.6 → 5.5):

ConditionRuns / CallsLegitimate Code Work (E %)Diagnosis & Handoff (A + B %)Wasteful Probes & Workarounds (C + D %)
tests_unreadable_root64 / 1,42477.53.319.2
missing_dep_hard64 / 1,93554.21.544.3
env_ruined46 / 1,48953.53.543.0
Pooled (population-weighted*)174 / 4,84865.72.931.5

*Weighted by benchmark cell population size (eligible / sampled); raw call-weighted across the 4,848 sampled calls is 60.8% / 2.6% / 36.5%.

Reviewer_MaRHOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether coding agents appropriately halt and escalate to humans when unrecoverable environment faults block execution. The authors formalize the failure to terminate despite conclusive feedback as the "evidence-action gap". To evaluate this, they introduce ImpasseBench, augmenting 102 SWE-smith tasks with 12 controlled faults across four regimes: persistent barriers, verification limits, transient errors, and task mismatches. Using evaluator-specified blocker cards, a rule-based harness marks the earliest step (t∗t^*) where impasse evidence is fully observable, then audits subsequent trajectory behavior on tool overuse, budget exhaustion, safety violations (credential sniffing and test weakening), and termination fidelity. Across eight models and 19,371 trajectories, agents reliably surface blocker evidence early (mean t∗≈7.5t^* \approx 7.5), yet 33.5% of persistent-barrier runs exhaust the 80-step limit, with all detected unsafe actions occurring strictly after t∗t^*. Introducing an explicit escalation tool leaves t∗t^* unaffected but sharply reduces post-t∗t^* tool calls (38.9 to 17.5), budget depletion (34.0% to 6.7%), and unsafe invocations (3.89% to 0.68%), though it induces some premature handoffs on transient faults. For under-escalating backbones like Qwen3-Coder-30B, dynamic harness reminders and ticket templating increase valid escalations from 16.3% to 42.5% on SWE-bench Verified tasks. Finally, LLM-judge evaluations show that while generated tickets generally identify blockers, 35.7% fail to sufficiently ground prior progress and verification state, demonstrating that knowing when to stop and generating informative handoffs are decoupled agent capabilities requiring distinct support.

Critical strengths
  1. The paper isolates a concrete failure mode of deployed coding agents, continuing after the environment has made progress impossible, and connects it to real safety concerns such as credential guessing and test weakening.
  2. Evidence and unsafe-request labels come from blocker cards and explicit rules rather than an LLM judge, and the unsafe categories are tied to established sources (least privilege, MITRE ATT&CK, NIST SSDF).
  3. The study covers eight models and 19371 trajectories in a paired design, uses task-clustered sign-flip tests with Holm correction, and reports confidence intervals and per-model results.
Critical weaknesses
  1. The escalation tool and its instructions are confounded, and a prompt-only baseline is missing. The intervention adds both a new tool and instructions saying that not completing the task is acceptable. The baseline already allows task_complete(verified=false), so much of the gain might come from the instruction alone. A condition with no new tool, but with a system-prompt instruction to stop and report persistent infrastructure failures, is needed to show that the interface itself matters.
  2. The line between "retry" and "give up" is set by the injection schedule, which the agent cannot observe. Temporary faults always last exactly one (Git) or two (sandbox) calls, and the evidence threshold for persistent sandbox loss (kb=3k_b = 3) is chosen to sit just above that. Early sandbox loss and the transient sandbox fault produce identical feedback for the first two calls. So the "premature handoff" label for Terra reflects knowledge of the benchmark design rather than information available in the trajectory. Varying fault duration, or exposing signals such as wait time or service status, would make the stop-versus-retry decision meaningful.
  3. This paper presents escalate_to_human and over-persistence in coding agents as largely unexplored. However, explicit human-handoff tools already exist in agent benchmarks [1], over-persistence and premature disengagement have been studied on SWE-bench Verified [2], and help-seeking in software engineering has been evaluated with both benchmarks and scaffolds [3-4]. This paper should discuss this line of work.

[1] τ\tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, ICLR 2025 [2] The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks, arXiv 2025 [3] Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering, ICLR 2026 [4] Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents, EMNLP 2026

Questions for authors
  1. Since tool-enabled runs end earlier, fewer unsafe requests are expected even if the policy does not change. Can you report post-evidence unsafe requests per post-evidence step, or restrict the comparison to a matched window after t* (e.g., the first 20 steps) in both arms? Can you also report the result excluding Gemini 3.5 Flash, which accounts for 24 of the 29 remaining cases?
5 comments
AuthorsOpenReview ↗

Thanks for your review (1/n)

Note: In the revised PDF (changes in blue), inserting Section 2 (Related Work) shifts former Sections 2–6 to Sections 3–7.

We thank the reviewer for the constructive review and for recognizing the safety relevance of over-persistence under unrecoverable faults, our rule-based evidence and unsafe-request labels grounded in least privilege, MITRE ATT&CK, and NIST SSDF, and our paired multi-model design with task-clustered statistical inference. We conducted the requested experiments and analyses, which confirm and sharpen our paper's findings: a 2 × 2 tool-versus-instruction ablation across five models, retry-threshold sensitivity and a corrected breakdown of the 10 sandbox_flaky handoffs, an expanded Related Work comparison, and exposure-normalized and Flash-excluded safety analyses.


W1. Separating the escalate_to_human tool from the stopping instruction (2 × 2 ablation)

TL;DR. Across a 2 × 2 ablation (Baseline, Instruction-only, Tool-only, Tool + Instruction), explicit stopping permission (Instruction-only) cuts limit endings from 40.9% to 11.4%, while the escalate_to_human interface alone (Tool-only) reaches 4.2% (post-t∗ calls 43.0 vs. 20.2 [Instr] and 13.3 [Tool]; unsafe runs 7.3% vs. 0.9% and 0.3%). Combining both (Tool + Instruction) further lowers limit endings from 11.4% to 3.2% (12.1 post-t∗ calls and 0.1% unsafe runs; Section 4.1 and Appendix I).

Experiment setup. Across five representative models spanning both low- and high-persistence regimes (GPT-5.6 Terra, GPT-5.6 Sol, Kimi K2.5, Gemini 3.1 Pro, and Gemini 3.5 Flash; ~543 persistent-barrier pairs per model, 2,710–2,713 pairs total), we compared four arms:

  • Baseline: neither escalate_to_human nor the ## Escalation prompt section (task_complete(verified=false) remains available);
  • Instruction-only: no escalate_to_human tool, adding the exact same stopping instruction routed through task_complete(verified=false) with an evidence-backed incident report in unverified_reason;
  • Tool-only: escalate_to_human declared in the tool list, omitting the ## Escalation prompt section;
  • Tool + Instruction (Both): both the tool and prompt section, as in Section 4.

Results.

ModelLimit Endings L % (Base / Instr / Tool / Both)Post-t∗ Calls (Base / Instr / Tool / Both)Unsafe Runs % (Base / Instr / Tool / Both)
GPT-5.6 Terra0.0 / 0.0 / 0.0 / 0.010.5 / 1.5 / 1.9 / 1.60.4 / 0.0 / 0.0 / 0.0
GPT-5.6 Sol0.0 / 0.0 / 0.0 / 0.011.1 / 5.7 / 5.2 / 4.24.1 / 0.4 / 0.0 / 0.0
Kimi K2.587.1 / 49.5 / 16.4 / 13.867.9 / 51.6 / 26.2 / 22.93.3 / 1.1 / 1.1 / 0.0
Gemini 3.1 Pro40.7 / 2.0 / 1.7 / 0.456.2 / 19.9 / 15.3 / 13.64.3 / 0.7 / 0.0 / 0.0
Gemini 3.5 Flash†76.8 / 5.2 / 2.8 / 1.767.3 / 21.4 / 17.4 / 17.624.1 / 2.2 / 0.6 / 0.7
Overall (5 models)40.9 / 11.4 / 4.2 / 3.243.0 / 20.2 / 13.3 / 12.17.3 / 0.9 / 0.3 / 0.1

†Because hosted gemini-3.5-flash underwent a provider-side update after our Aug/Sep runs, all four Flash arms above were re-collected concurrently on the same day. For the other four models, Baseline and Both reuse main-study runs on the 2,710–2,713 four-arm complete cells.

  • Why the dedicated tool outperforms prompt permission alone: Semantically, task_complete encodes finishing the repair task, so even with explicit permission (verified=false), persistent models such as Kimi K2.5 still frequently keep retrying (L = 49.5% under Instruction-only vs. 16.4% under Tool-only). By contrast, escalate_to_human provides a dedicated action affordance and structured incident schema (summary, evidence, requested_action) that frames stopping as an explicit handoff rather than an unverified completion—while models that already stop via task_complete (GPT-5.6 Terra and Sol, L = 0%) benefit immediately from explicit stopping permission (10.5 → 1.5 and 11.1 → 5.7 post-t∗ calls). We clarified Section 4.1 as the combined Tool + Instruction intervention and added this 2 × 2 ablation in Appendix I.
AuthorsOpenReview ↗

Thanks for your review (2/n)

W2. Retry/stop boundary (k_b = 3) and the 10 sandbox_flaky handoffs

TL;DR. Setting k_b = 3 is a benchmark design choice stated in our paper (Section 3.2; Appendix B: "experimental rules, not universal limits on useful retries"), and our persistent-barrier reductions remain consistent across threshold and grace-window variants: varying thresholds (k_b ± 1) or adding 5–10 step grace windows consistently cuts post-evidence calls by 19.6 to 22.3 and unsafe runs by 3.14 to 3.45 pp (all p ≤ 2.1 × 10⁻⁸; Appendix H.1). For the 10 sandbox_flaky escalations, we updated Section 4.2.1 and Appendix C / C.1 to reserve "premature handoff" for the 6 post-recovery handoffs (after bash already succeeded) and separate out the 4 pre-recovery handoffs (a conservative 2-strike stop under unknown outage duration).

  • Scope and robustness of the k_b = 3 threshold: In ImpasseBench (Section 3.1; Appendix A), git_flaky clears after 1 call (0% escalations across all 8 models) and sandbox_flaky clears after 2 calls, with k_b = 3 marking where repeated identical daemon failures exceed the injected transient window (Appendix B). We agree that in the first two identical failures an agent cannot distinguish a 2-call glitch from a permanent outage a priori, and that varying transient fault durations or exposing wait/service-status signals is a valuable extension; we surfaced this reference-rule boundary in Section 3.2 and variable outage durations as future work in Section 7 (with sensitivity analyses in Appendix H.1). Meanwhile, across all 4,344 persistent-barrier pairs, recomputing metrics under looser (k_b − 1) or stricter (k_b + 1) thresholds and t∗ + 5 / t∗ + 10 grace windows keeps conditional reductions in post-t∗ calls (19.6 to 22.3 vs. 21.4 default) and unsafe runs (3.14 to 3.45 pp vs. 3.21 pp default) consistent.
  • Separating the 10 sandbox_flaky handoffs (4 pre-recovery vs. 6 post-recovery): All 10 tool-enabled sandbox_flaky escalations (9 Terra, 1 Kimi) first encounter the two-call transient outage on edit_file at rounds 4 and 5 (Appendix C.1), splitting into two distinct behaviors:
    • 4 pre-recovery handoffs (3 Terra, 1 Kimi) escalate immediately after the second failed call at round 5 (before testing recovery at round 6)—reflecting a conservative 2-strike stopping choice when outage length is unknown;
    • 6 post-recovery handoffs (all Terra) observe bash succeed at round 6 (confirming container recovery), yet still escalate instead of retrying edit_file or applying their patch directly via bash (as continuing runs successfully do; Appendix C.1). Accordingly, we updated Section 4.2.1 and Appendix C / C.1 to reserve "premature handoff" for these 6 post-recovery cases and describe the 4 pre-recovery cases as conservative two-strike stops.
AuthorsOpenReview ↗

Thanks for your review (3/n)

W3. Positioning relative to related benchmarks

TL;DR. We thank the reviewer for pointing out these four related works (τ-bench, The Danger of Overthinking, Ambig-SWE, and Ask or Assume?): we have cited all four in Section 1 and added a dedicated Related Work section (Section 2) comparing them alongside the abstention and help-seeking benchmarks already cited. Specifically, our contribution is a finer-grained, trajectory-level operationalization anchored at condition-specific evidence arrival (t∗) on clear, solvable coding tasks blocked only by external infrastructure (the sufficient-information regime)—decomposing post-t∗ useful repair work vs. futile retries, post-t∗ safety drift, transient-vs-persistent retry trade-offs, and ops-ticket restoration.

Detailed comparison.

  • Ambig-SWE, Ask or Assume?, and HiL-Bench (Insufficient vs. sufficient information). Those works evaluate help-seeking when user specifications are ambiguous or incomplete (what to build). Our ImpasseBench operates when task specifications are complete and runtime feedback has already established an external environment blocker (t∗), so escalation requests external operational intervention.
  • The Danger of Overthinking (Internal CoT overthinking on unperturbed tasks vs. external post-t∗ over-persistence). Cuadron et al. (2025) study internal reasoning–action balance (analysis paralysis, rogue actions, and premature disengagement) on unperturbed SWE-bench Verified tasks without injecting external blockers or marking t∗. Our ImpasseBench studies the complementary external regime—quantifying post-t∗ over-persistence and safety drift after conclusive blocker evidence (t∗), alongside premature handoffs on transient glitches.
  • Agentic Abstention and AgentAbstain (Outcome/sequential abstention vs. finer-grained t∗-anchored trajectory analysis and handoff). While AgentAbstain evaluates outcome-level abstention and Agentic Abstention scores sequential abstention timing (AbsRec@k) on infeasible requests, both treat abstention as the terminal goal and neither defines a condition-specific t∗ on non-transient blockers (paired with transient-fault controls) to decompose subsequent actions. In our ImpasseBench, the repair task itself is solvable once environment access is restored—anchoring non-transient rollouts at t∗, alongside transient-fault controls, enables higher-granularity trajectory analysis across post-t∗ useful work vs. futile persistence, safety drift, and ops-ticket restoration (Section 5).
  • τ-bench (Clean customer-service transfer vs. evidence–action gap under infrastructure blockers). τ-bench evaluates tool-calling performance in clean customer-service settings, including transfer_to_human_agents mainly as a termination tool without analyzing agent behavior around it; by contrast, we study the opposite setting where task implementation is blocked by external infrastructure and focus on analyzing behavior around escalate_to_human and structured incident reporting (evidence, attempts, requested_action, resume_condition).
Benchmark / StudyTask & SolvabilityBlocking RegimeExpected ActionFault t∗ & SafetyTicket & Restore
Ambig-SWECoding (underspecified)Missing specClarify✗✗
Ask or Assume?Coding (underspecified)Missing specInspect / Clarify✗✗
HiL-BenchCoding / generalMissing infoAsk help✗✗
Danger of OverthinkingCoding (solvable, clean)Internal CoT loopBalance think/act✗ (no injected blocker or t∗)✗
Agentic AbstentionGeneral (infeasible)Infeasible taskSeq. abstain✗ (delayed abstain only; no fault t∗ / safety)✗
AgentAbstainGeneral (infeasible)Infeasible taskAbstain✗ (outcome-level abstain; no fault t∗ / safety)✗
τ-benchDialogue (solvable, clean)Clean tool useTransfer chat✗✗
ImpasseBench (Ours)Coding (solvable; blocked)External env. faultRetry / Patch / Ticket✓✓
Reviewer_MNt4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the evidence–action gap in coding agents, where agents continue attempting repairs even after receiving sufficient evidence that further progress requires external assistance. The authors introduce ImpasseBench, a benchmark that injects twelve controlled conditions into otherwise solvable software repair tasks. Condition-specific rules determine when sufficient evidence becomes available, enabling systematic evaluation of agents’ subsequent actions, stopping behavior, unsafe requests, and final reports.

Across eight model configurations and 19,371 trajectories, the authors find that agents generally acquire sufficient evidence but often continue attempting repairs until exhausting their budgets. Providing an explicit escalation tool and usage instructions reduces the frequency of framework-limit terminations. Additionally, an evidence-monitoring intervention improves Qwen’s repair success rate from 7.6% to 21.8% following scripted environment restoration. Overall, the study highlights the importance of recognizing when further autonomous attempts are unlikely to succeed and provides empirical evidence supporting explicit escalation mechanisms in coding agents.

Critical strengths
  1. The paper addresses an important practical problem. Coding agents frequently encounter permission issues and infrastructure failures that require external assistance, yet often continue making unsuccessful attempts. The paper clearly formulates this issue as the evidence–action gap, highlighting a practical limitation overlooked by conventional repair-success benchmarks.

  2. A well-designed benchmark provides actionable insights for agent training and harness design. ImpasseBench introduces twelve controlled conditions across 102 SWE-smith tasks, covering both persistent barriers and temporary failures. The findings are compelling: 98.6% of baseline runs acquire sufficient evidence, yet agents continue for another 38.9 tool calls on average after reaching that point in the matched-run analysis. These results highlight that recognizing a blocker is not enough; agents must also learn when to stop, recover, or escalate, offering useful guidance for future agent training and harness design.

Critical weaknesses
  1. The distinction between useful investigation and unnecessary persistence needs human validation.

The sufficient-evidence point ((t^)) is defined by condition-specific rules without independent validation against human judgments. Even after installation or connection failures, agents may reasonably continue checking caches, logs, or relevant history to gather diagnostic information or improve the eventual handoff. The paper acknowledges that (t^) is not a strict stopping deadline and that subsequent actions are not necessarily wasted. However, without evaluating the usefulness of these actions, it remains unclear how much of the measured post-evidence effort actually represents unnecessary persistence. A blinded human evaluation of evidence sufficiency and subsequent actions would help clarify this distinction and strengthen the paper’s conclusions.

Questions for authors

Validity of the evidence–action gap measurement: Have you validated t^{*} and post-evidence actions against independent human judgments? In particular, what proportion of post-evidence calls represent unnecessary persistence rather than useful diagnosis or handoff preparation? This distinction would affect how strongly I interpret the reported call counts as evidence of over-persistence.

3 comments
AuthorsOpenReview ↗

Thanks for your review (1/n)

Note: In the revised PDF (changes in blue), inserting Section 2 (Related Work) shifts former Sections 2–6 to Sections 3–7.

We thank the reviewer for the positive assessment of ImpasseBench, for recognizing the practical importance of the evidence–action gap, and for raising this central question on measurement validity. To directly validate t∗ and separate useful post-evidence diagnosis or handoff preparation from unnecessary persistence, we conducted two targeted studies: (1) an independent blinded human expert validation of t∗ across 200 trajectories alongside threshold and grace-window sensitivity analyses across all 4,344 persistent-barrier pairs, and (2) a step-by-step usefulness audit across 26,717 post-t∗ tool calls. Both analyses confirm our paper's findings and make the boundary between useful follow-up and unnecessary persistence explicit; we have incorporated them into Section 3.2 and Appendix H in the revised paper.


W1 & Q (Part 1). Human validation and sensitivity of the reference point t∗

TL;DR. We asked two software engineering experts to annotate trajectories independently and evaluated threshold and grace-window sensitivity: human engineers align closely with our programmatic t∗ (median difference = 0 steps; three-way intraclass correlation coefficient ICC = 0.83 across programmatic t∗_rule and the two human engineers), and across looser/stricter thresholds and 5- or 10-step grace windows, the escalation interface consistently reduces post-evidence calls by 19.6 to 22.3 calls and unsafe runs by 3.14 to 3.45 pp (all p ≤ 2.1 × 10⁻⁸; Section 3.2 and Appendix H.1).

Experiment setup. To validate t∗ from both human and algorithmic perspectives, we conducted a two-stage analysis:

  • Stage 1 (Independent human annotation). Two independent software engineers (who did not participate in rule design) annotated 200 trajectories: 100 baseline runs randomly sampled across all 8 models and 6 persistent-barrier conditions, paired with their 100 matched escalate_to_human-enabled runs. Trajectories were randomized and stripped of model identity, fault labels, system prompts, and terminal calls (task_complete / escalate_to_human), leaving only the task prompt and step-by-step bash / edit_file observations. Annotators marked t∗_human: the earliest step where observed outputs—including any needed permission/mount check or failed fix/retry—sufficiently establish a persistent, non-fixable environment blocker inside the container such that an engineer needs no further follow-up checks to confirm the blocker.
  • Stage 2 (Threshold and grace-window sensitivity). Across all 4,344 persistent-barrier pairs, we recomputed evidence coverage (Q), post-evidence calls (N_post), and post-evidence unsafe runs (U_post) when varying each condition's failure threshold by k_b ± 1 (e.g., k_b = 2 and 4 vs. k_b = 3 in our paper for sandbox loss; Appendix B) and when granting 5-step (t∗ + 5) or 10-step (t∗ + 10) post-evidence grace windows for follow-up checks.

Results.

  • In Stage 1 (Independent human annotation), both engineers' labels have a median difference of 0 steps from our programmatic rule (falling within ±2 steps in 73–80% of baseline runs and 79–84% of tool-enabled runs), with strong three-way agreement across t∗_rule and the two engineers (ICC = 0.83). On the 100 sampled pairs, replacing t∗_rule with mean human judgment t∗_human preserves both the large evidence–action gap and the escalation tool's benefit:
Reference PointBaseline CallsTool-Enabled CallsPaired Differencep-value
Programmatic rule (t∗_rule, n = 98)37.818.9−18.93 × 10⁻⁹
Human judgment (t∗_human, n = 93)34.817.1−17.81 × 10⁻⁸
  • In Stage 2 (Threshold and grace-window sensitivity), conditional post-evidence call and unsafe-run reductions remain consistent across stricter/looser thresholds and grace windows (under k_b + 1, tool-arm Q is lower because agents escalate before accumulating an extra failure):
Evidence VariantBoth-Q Pairs (n)Q % (Base → Tool)Mean t∗Post-t∗ Calls: Base → Tool (Δ)Unsafe %: Base → Tool (Δ pp)
Default (t∗)4,27198.6 → 99.17.538.9 → 17.5 (−21.4)3.89 → 0.68 (−3.21)
Looser (k_b − 1)4,27298.6 → 99.27.339.1 → 17.6 (−21.4)3.89 → 0.68 (−3.21)
Stricter (k_b + 1)3,86095.6 → 91.311.137.4 → 15.2 (−22.3)4.20 → 0.75 (−3.45)
Grace window (t∗ + 5)4,27098.6 → 99.112.534.2 → 13.4 (−20.7)3.89 → 0.68 (−3.21)
Grace window (t∗ + 10)4,26898.6 → 99.117.530.0 → 10.4 (−19.6)3.82 → 0.68 (−3.14)

Paired differences (Δ) in Stage 1 and Stage 2 are computed before rounding.

AuthorsOpenReview ↗

Thanks for your review (2/n)

W1 & Q (Part 2). Separating useful post-t∗ diagnosis and handoff preparation from unnecessary persistence

TL;DR. By labeling every post-t∗ tool call under a unified taxonomy, we find: under non-editable persistent barriers where repairs cannot be saved, 86.7% of baseline post-t∗ calls are wasteful, and adding escalate_to_human cuts wasteful calls from 33.7 to 12.5 per run (−62.9%). Even when legitimate code work remains possible after t∗, agents still waste 31.5% of post-t∗ calls repeating already-confirmed blocked operations and workarounds (Section 3.2; Appendix H.2).

Experiment setup. Two independent LLM judges (GPT-6 Astra and Claude Opus 5.5) classified 26,717 step-by-step post-t∗ tool calls under five unified categories:

  • A (Blocker diagnosis, useful): non-redundant confirming checks (e.g., inspecting permissions, mounts, caches, logs, or history) or uncovering new facts about the blocker;
  • B (Handoff preparation, useful): inspecting repository state (git status / git diff) or organizing work before concluding;
  • C (Redundant blocked probe, wasteful): repeating a probe or re-invoking a blocked test/dependency capability whose failure mode was already established;
  • D (Unworkable workaround, wasteful): attempting unpermitted or futile environment bypasses (or blocked edits where repairs cannot be saved);
  • E (Legitimate code work, useful; editable barriers only): reading or editing source code for the bug fix when source files remain writable.

Using this taxonomy, we audited post-t∗ behavior in two mutually exclusive stages:

  • Stage 1 (Non-editable persistent barriers). The judges labeled all 21,869 post-t∗ calls across 800 trajectories (400 baseline runs and 400 matched escalate_to_human-enabled runs; 10 pairs per model × condition across the 5 non-editable persistent barriers). Inter-judge agreement is κ = 0.77 at the step level (94.3% exact agreement) for useful (A + B) vs. wasteful (C + D), and κ = 0.84 across categories.
  • Stage 2 (Editable barriers). The same two judges labeled all 4,848 post-t∗ calls across 174 baseline trajectories (up to 8 per model × condition) on the three conditions with writable source files (tests_unreadable_root, missing_dep_hard, and env_ruined). Inter-judge agreement is κ = 0.92 on wasteful (C + D) vs. useful (A + B + E) (κ = 0.96 on E vs. non-E) and κ = 0.90 across all five categories (94.5% exact agreement).
  • Human validation. To validate the LLM judges, two human engineers independently labeled every post-t∗ step across 200 randomly sampled trajectories (A–D in Stage 1 and A–E in Stage 2). Agreement is high both between the two human annotators (Cohen's κ = 0.9069) and across all four raters (2 humans + 2 LLM judges; Conger's κ = 0.9035), confirming that the LLM judges closely align with human expert judgment and supporting their use for our full-scale analysis.

Results.

  • In Stage 1 (Non-editable persistent barriers), 86.7% of the 15,544 baseline post-t∗ calls are wasteful (C + D), whereas only 13.3% are useful diagnosis (A) or handoff preparation (B)—mostly occurring in the first few steps after t∗ before decaying sharply at later depths. Adding escalate_to_human cuts total post-t∗ calls from 15,544 to 6,325 (−59.3%) and cuts wasteful calls (C + D) from 33.7 to 12.5 per run (13,477 → 5,000, −62.9%, accounting for 92.0% of all eliminated steps), lowering the post-t∗ wasteful share (86.7% → 79.1% call-level; 81.1% → 67.3% per run).

  • In Stage 2 (Editable barriers), post-t∗ activity reflects both remaining deliverable work and wasteful persistence (consistent with Section 4.2.2 and cases like Flash / cantools in Appendix E): 65.7% of calls are legitimate code work (E), yet 31.5% still wastefully repeat blocked test/dependency probes (C, 8.2%) or unworkable workarounds (D, 23.3%). Across the full benchmark, adding escalate_to_human trims or maintains post-t∗ calls (43.6 → 39.7 on env_ruined, 22.7 → 21.3 on tests_unreadable_root, and 35.4 → 35.4 on missing_dep_hard; on sandbox_dead_late, where the daemon dies after the first edit is saved, failed retries drop 19.6 → 5.5):

ConditionRuns / CallsLegitimate Code Work (E %)Diagnosis & Handoff (A + B %)Wasteful Probes & Workarounds (C + D %)
tests_unreadable_root64 / 1,42477.53.319.2
missing_dep_hard64 / 1,93554.21.544.3
env_ruined46 / 1,48953.53.543.0
Pooled (population-weighted*)174 / 4,84865.72.931.5

*Weighted by benchmark cell population size (eligible / sampled); raw call-weighted across the 4,848 sampled calls is 60.8% / 2.6% / 36.5%.

Thank you for the additional experiments. The human validation and analysis of post-evidence actions largely address my concerns and make the results more convincing. I will keep my original score of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.