acceptodds
Withdrawn from ICLR 2027

The Repair Is Not the Task: Execution Records and False Completion in Agent Recovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When a tool call fails after a partial write, recovery must both restore a consistent state and complete the original task. Measuring how a correct repair's execution record shapes the rest of the task requires holding the repair and its successor state fixed. We show that once a compensating repair succeeds, its record can end recovery early, with the agent behaving as if the repair's postcondition were the task's; we call this failure repair-as-completion. We score every executed step against the authorized goal under an explicit execution contract, keep repair and task completion as separate outcomes, and continue one executed step under records that differ in nothing else. On a two-resource reservation task, the most recent record raises Qwen3-14B's safe completion from 2 to 33 of 40 completable units. After a compensation it ends the task: on 18 retail orders adapted from τ²-bench, Qwen3-14B completes all 18 from the current state alone and, with the record, declares all 18 complete while every order is still pending; after a forward repair, or a readback that leaves the same state, it proceeds with the task. Six of 27 models, each run with thinking off, stop on at least 6 more of the 18 orders with the record, four of them specifically after the repair, and DeepSeek-V4.1-Flash also stops on a file-log task and on telecom lines. Under an exact completion gate before finish that states why it rejects, 100 of the 159 episodes that otherwise end in false completion finish safely, and offline Qwen3-32B verdicts on the stored prompts reproduce the same gated outcomes while accepting all 108 reached-state presentations. Restating the goal lowers the stop to at most 2 of 18 orders for Qwen3-14B and Qwen3.5-9B. Recovery benchmarks should report repair success and task completion as separate outcomes.

Primary area:
datasets and benchmarks
Keywords:
tool-using agents, failure recovery, false completion, execution records, execution contracts, agent evaluation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KVocOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors discuss a very specific failure mode of AI agents: Give them a task, the execution fails, the agent repairs the failed operation. At this point the authors step in and analyze a decision: Does the agent (A) continue to retry the original task, (B) declare the successful repair a successful completion. The authors compare continuations from a repaired state, but intervene on whether a record of the repair is present. In some constellations, this record can prompt the model to declare success prematurely.

Critical strengths

The problem of agent runtimes, repairing failed operations, retry, etc is highly relevant. There is a plethora of papers that proposes an all-encompassing solution, i.e. some form of agent runtime or orchestration mechanism that is supposed to do away with failures for good. I find it refreshing to see a more narrowly scoped analysis of one specific mechanism.

Critical weaknesses

Unfortunately I also believe that the narrow scope of the analysis is also the biggest weakness of the paper. The mechanism that the authors present: "Agent makes a mistake, repairs the mistake, then is so satisfied with the successful repair that it declares 'My job here is done'" feels somewhat contrived. On first reading I couldn't help thinking "This feels like a prompting artifact". I have mentioned under strengths that other works present turnkey solutions for identifying and repairing errors and then retrying, based on all kinds of systems. And that I find it refreshing to see a narrowly scoped error analysis. But it would be important to show that the error is indeed still relevant. There are all kinds of systems out there that should trivially prevent this failure mode. On some level the error analysis here feels like the authors set up an experiment that does "just enough" agentic AI to let the model fail in a weird way, but "not enough" to actually let it use the tools and their records in a reasonable way, i.e. prevent it from failing. As far as I understand the experiments are based on a peculiar interface: Each decision is a separate single-turn request, thinking is disabled, and the models studied are relatively small. The authors do include native multi-turn tool calling (Table 17), and it seems like now everything is safe? At this point, the entire headline failure feels like something that the authors have conjured up themselves, by putting an LLM into a contrived error prone harness. This motivates voting for rejection. I don't think the studied error is at all relevant and any generalizable insights are being generated.

There are two additional weaknesses:

The related works feel too superficial. Just a single note here, this feels highly relevant: Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents (2026) I have the impression that the presentation of related works is somewhat misaligned with the present manuscript. It cites very seminal papers, such as SagaLLM, but no closely related mechanisms. Which do exist I think (see the one paper I've mentioned, and I'm convinced that doing an in-depth search on premature termination, misinterpreting completion signals, etc should be studied). This is meant as feedback, it has not influenced my final rating!

The failure analysis is weak. Choosing finish is a violation of the execution contract, of the overall goal to be accomplished. But it seems that models are able to realize that they are not done. (Table 26). This is probably why native multi-turn tool calling doesn't exhibit these failures. And it does throw some serious doubts on your whole methodology (switching thinking off...) It seems that you've set the models up for failure. As I said before: I don't think any principled insights are generated here, the conclusions seem contrived. Similarly I find the mitigation contribution limited. Why do you need to mitigate something that doesn't happen if you use the models in the intended way (native multi-turn tool calling)? The whole error analysis + mitigation story reads: We've created a harness that no one should use, it fails in a weird way, now here's how to fix this failure.

Questions for authors

Are you sure you've appropriately covered related works?

Reviewer_bewROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies "failure repair-as-completion" , where an agent incorrectly treats a successful compensating repair as completion of the original task. When a tool call fails after a partial write, recovery must both restore a consistent state and complete the original task.

The paper propose an evaluation framework based on an execution contracts that separates repair success from task completion. The paper conduct controlled experiments by holding the environment state fixed and varying the execution records provided to the agent. The evaluation covers retail orders adapted from τ²-bench, telecom lines, file-log tasks, and two-resource reservation tasks across multiple LLMs.

The results show that execution records help the first step after a rejected call and cut the second short once a compensation succeeds: six of 27 models, each run with thinking off, declare pending retail orders complete, four specifically after the repair. Stating the task’s status nearly removes the stop for Qwen3-14B and Qwen3.5-9B, and a completion gate before finish returns most blocked episodes to a completed task. Recovery evaluation should report whether the task, not only the repair, is complete

Critical strengths
  • The paper points out an interesting failure mode.
  • Comprehensive evaluation. The experiments cover multiple models, and provide detailed instructions for reproducibility.
Critical weaknesses
  • The experiments mainly rely on artificially created partial failure scenarios, such as partial order refunds. Such issues are typically handled via deterministic mechanisms like database transactions. The paper does not discuss why an LLM agent should be directly responsible for these operations, and whether these patterns will appear in real workflows.

  • Limited technical contribution. The proposed mitigation method mainly rely on prompt-based task-status reminders and a completion gate. Its difference from existing methods in runtime verification or task success checking is not clear. No analysis on efficiency as well.

Questions for authors
  1. In what types of real-world agent workflows, is failure recovery required to be performed by an LLM agent rather than being handled through traditional transactional database mechanisms or deterministic logic? Could you provide some real-world examples?

  2. How does model scale, reasoning efforts , or prompt formatting influence the results?

  3. What are the novel technical contributions of the proposed method , compared to existing task verification methods?

Reviewer_DX5MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies “repair-as-completion”: an agent correctly undoes a partially executed operation but then treats that repair as completion of the original request. The authors isolate the effect of execution records by continuing from the same repaired state with different records, while keeping the goal and execution contract fixed. Experiments cover retail cancellation, file-log, telecom, and reservation tasks across multiple models. The clearest result is that Qwen3-14B completes all 18 retail orders from the state alone but falsely declares all 18 complete when shown the compensation record. The paper also evaluates goal reminders and completion gates, arguing that recovery benchmarks should score repair success separately from task completion.

Critical strengths

The experimental design isolates the effect of the record convincingly. Keeping the repaired state, goal, and budget fixed makes the main comparison easy to interpret. The same-state readback control and record-element ablations further show that, for several models, the effect depends on how the repair is reported rather than merely on adding context.

The paper makes a useful distinction between successful repair and successful task execution. Scoring executed effects separately from completion claims exposes failures that a first-step recovery metric would miss. The completion-gate experiments also go beyond detecting false claims: 100 of 159 otherwise falsely completed episodes su

Critical weaknesses

Generalization remains limited despite the large number of experimental conditions. The 18 retail orders share one structural template, and most follow-up experiments reuse the same worlds. The effect also depends strongly on the interface and reasoning mode: Qwen3-14B and Qwen3.5-9B complete every retail order under native tool calling, while thinking substantially reduces the failures. These controls are valuable and transparently reported, but they leave uncertainty about how often the phenomenon matters in realistic agent workflows.

The practical evidence for the completion gate is narrower than the headline result suggests. Trajectories are collected using an exact executor-based gate, with the model checker evaluated offline on the resulting prompts. This supports reconstruction of those outcomes, but successful-state checks use scripted presentations, and the comparison does not match total inference cost. Table 28 also shows substantial false rejection on some request-only telecom conditions. The gate is promising, but its reliability when completion must be inferred from ordinary task history remains less established.

Questions for authors
  1. What explains Qwen3-14B’s change from 18/18 false completions in the single-turn setting to none under native tool calling? Defining the actions alone does not remove its single-turn effect, so which remaining interface difference appears responsible?

  2. Could you evaluate the checker without the structured authorized_goal while retaining the pre-attempt observation? Table 28 removes both, making it difficult to distinguish dependence on an explicit endpoint from lack of evidence needed to verify the requested change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.