- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors discuss a very specific failure mode of AI agents: Give them a task, the execution fails, the agent repairs the failed operation. At this point the authors step in and analyze a decision: Does the agent (A) continue to retry the original task, (B) declare the successful repair a successful completion. The authors compare continuations from a repaired state, but intervene on whether a record of the repair is present. In some constellations, this record can prompt the model to declare success prematurely.
Critical strengths
The problem of agent runtimes, repairing failed operations, retry, etc is highly relevant. There is a plethora of papers that proposes an all-encompassing solution, i.e. some form of agent runtime or orchestration mechanism that is supposed to do away with failures for good. I find it refreshing to see a more narrowly scoped analysis of one specific mechanism.
Critical weaknesses
Unfortunately I also believe that the narrow scope of the analysis is also the biggest weakness of the paper. The mechanism that the authors present: "Agent makes a mistake, repairs the mistake, then is so satisfied with the successful repair that it declares 'My job here is done'" feels somewhat contrived. On first reading I couldn't help thinking "This feels like a prompting artifact". I have mentioned under strengths that other works present turnkey solutions for identifying and repairing errors and then retrying, based on all kinds of systems. And that I find it refreshing to see a narrowly scoped error analysis. But it would be important to show that the error is indeed still relevant. There are all kinds of systems out there that should trivially prevent this failure mode. On some level the error analysis here feels like the authors set up an experiment that does "just enough" agentic AI to let the model fail in a weird way, but "not enough" to actually let it use the tools and their records in a reasonable way, i.e. prevent it from failing. As far as I understand the experiments are based on a peculiar interface: Each decision is a separate single-turn request, thinking is disabled, and the models studied are relatively small. The authors do include native multi-turn tool calling (Table 17), and it seems like now everything is safe? At this point, the entire headline failure feels like something that the authors have conjured up themselves, by putting an LLM into a contrived error prone harness. This motivates voting for rejection. I don't think the studied error is at all relevant and any generalizable insights are being generated.
There are two additional weaknesses:
The related works feel too superficial. Just a single note here, this feels highly relevant: Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents (2026) I have the impression that the presentation of related works is somewhat misaligned with the present manuscript. It cites very seminal papers, such as SagaLLM, but no closely related mechanisms. Which do exist I think (see the one paper I've mentioned, and I'm convinced that doing an in-depth search on premature termination, misinterpreting completion signals, etc should be studied). This is meant as feedback, it has not influenced my final rating!
The failure analysis is weak. Choosing finish is a violation of the execution contract, of the overall goal to be accomplished. But it seems that models are able to realize that they are not done. (Table 26).
This is probably why native multi-turn tool calling doesn't exhibit these failures.
And it does throw some serious doubts on your whole methodology (switching thinking off...)
It seems that you've set the models up for failure. As I said before: I don't think any principled insights are generated here, the conclusions seem contrived.
Similarly I find the mitigation contribution limited. Why do you need to mitigate something that doesn't happen if you use the models in the intended way (native multi-turn tool calling)?
The whole error analysis + mitigation story reads: We've created a harness that no one should use, it fails in a weird way, now here's how to fix this failure.
Questions for authors
Are you sure you've appropriately covered related works?