acceptodds
Under review as a conference paper at ICLR 2027

When Is Self-Repair No Longer Possible? Certifying Irrecoverable Turning Points in Tool-Using Agents

Abstract

A failed repair attempt does not reveal whether a tool-using agent has exhausted every legal recovery path or merely selected a poor action. Existing evaluations often collapse environmental irrecoverability, resource insufficiency, recognition errors, planning errors, and execution errors into one end-to-end score. We introduce an evaluation framework for irrecoverable turning points that specifies the recovery goal, state, action catalogue, and resource budget as an executable contract. A replayable successful path certifies ecoverability, whereas a forward-closed goal-disjoint set certifies irrecoverability. Across 671 checkpoints in finite-state environments, every label agrees with independent search. Contract-sufficiency evaluation verifies 671 full certificates and 945 action-ablation certificates; hiding the remaining budget, capability flags, or domain state creates 9, 42, and 70 ambiguous observation groups. On 24 stateful AgentDojo and AppWorld task clusters, GPT-5.6 Sol, DeepSeek-V4.1-Flash, Qwen3-4B, and Claude Opus5.5 complete 96/96 real-tool repairs. Their exact turning-point results are 24/24, 23/23 among scorable cases, 0/24, and 24/24, respectively. A recognition–planning–execution decomposition further localizes failures concealed by end-to-end outcomes. The framework turns claims that an agent can no longer self-repair into scope-bounded, certificate-backed, and stage-diagnostic evidence for auditable stopping and escalation decisions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
tool-using agents, self-repair, irrecoverable turning point, reachability certificate, snapshot branching, stage diagnosis, resource budget

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5NtrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an evaluation framework for irrecoverable turning points in agent trajectories.

Critical strengths
  1. The paper studies poses important problem of distinguishing recoverable from irrecoverable failures, and also those recoverable under a fixed budget.
  2. The evaluation framework, such as the independent checker evaluation, is clean and nicely designed.
  3. Interesting finding that qwen3-4B is able to recover from recoverable failures but is not able to identify irrecoverable failures.
Critical weaknesses
  1. The paper is hard to follow because many terms and methods are used without proper definition, e.g., "certificate", "witness" etc. Please use consistent terminology and improve the flow of the paper.
  2. There aren't many useful findings, except the qwen behavior. Perhaps more smaller models need to be evaluated to show the benefits of the evaluation framework.
  3. It is unclear why model self-reported runtime is being compared. No one would plausibly use self-reports as resource certificates.
  4. The evaluation framework is useful but seems synthetic and does not analyze real agent trajectories. While reading the earlier sections, such as Equation 3, the paper gives the impression that the "turning point" from real agent trajectories would be evaluated but that is not the case.
  5. Only 24 task clusters are evaluated.
  6. Please highlight novelty compared to recent work around measuring irrecoverability in LLM agents, such as, [1].

[1] Ruan, K., Huang, Z., Zhou, Z., Wei, Q., Lin, J., Wang, X., & Sun, H. (2026). Doomed from the start: Early abort of llm agent episodes via a recall-controlled probe cascade. arXiv preprint arXiv:2607.06503.

Questions for authors
  1. How is the independent checker and certificate verification implemented?
  2. How were the irrecoverable environments created, i.e., what makes each c2 irrecoverable?
  3. Please explain the setup of experiments done in Section 5.4 clearly.
Reviewer_bAPpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes framework for assessing recoverability in tool-using AI agents. The framework formalizes recovery goals, available actions, environment states, and resource budgets to determine whether a valid recovery path exists and identify the critical action that makes recovery impossible. It further introduces recovery witnesses and irrecoverability certificates to support these judgments. The authors evaluate the framework through finite-state environments and real-tool experiments on AgentDojo and AppWorld. By decomposing agent performance into recognition, planning, and execution, the experiments reveal differences in recovery capabilities that conventional end-to-end success metrics may overlook. Overall, the paper provides a structured approach to evaluating and diagnosing recovery failures in tool-using agents.

Critical strengths

The problem is interesting and clearly motivated.

Critical weaknesses

The presentation of this paper needs to be significantly improved. Especially, the role of the model together with the developed theory need to be better tightened and clarified to fit the experiments, and potential broader applications.

The contributions and the conducted analysis are a bit weak, in terms that they do not form together a convincing result to prove the practicality of the designed method in realistic problems (as those can be done by agents).

Questions for authors
  1. The paper could be improved, after a throughout proofread. Polishing the presentation could also help.

  2. Despite that the problem is clearly motivated, would it be possible to make the contribution higher, by demonstrating how the designed evaluation framework can be used in practical problems?

Reviewer_atY5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines an irrecoverable turning point as the first action after which no legal recovery remains under an explicit recovery contract. It combines replayable recovery witnesses, finite-state irrecoverability certificates, and contract-sufficiency tests, with experiments in a custom environment and AgentDojo/AppWorld snapshots.

Critical strengths
  1. The paper clearly separates unsuccessful repair attempts from the absence of any legal recovery path. Positive witnesses and independently checked negative certificates provide an auditable evaluation framework (Sec. 3).

  2. The contract-sufficiency tests expose how labels depend on available actions and observable state, making otherwise implicit evaluation assumptions testable.

Critical weaknesses
  1. Real-environment certification is insufficiently documented in the main text. The certificate criterion applies to finite deterministic systems, while reported certificate counts come from the custom environment. For AgentDojo/AppWorld, the state abstraction, action catalogue, and completeness of outgoing transitions are unclear. These details are essential for distinguishing genuine irrecoverability from limitations imposed by the contract.

  2. The empirical study provides limited evidence for turning-point recognition. The closed-loop study uses only 24 clusters and achieves 96/96 repairs. Models classify two prebuilt snapshots rather than locate turning points in their own trajectories. Section 5.4 includes replay-based localization, but its connection to the evaluated models is unclear. Simple baselines and uncertainty estimates are also missing.

Questions for authors
  1. How are real-environment irrecoverability certificates constructed and checked? Do the labels remain valid under the environment’s full toolset?
  2. Can models locate turning points in their own trajectories, compared with an inverse-operation rule baseline and a full-trajectory judge?
  3. Can you provide harder, longer-horizon cases, repeated runs, and confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.