- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies recovery mechanisms for long-horizon LLM agents when execution trajectories become corrupted by previous mistakes. The authors argue that errors in long-horizon tasks often accumulate over time, causing agents to continue from an invalid state or repeat unsuccessful behaviors. Existing approaches mainly rely on continuing execution, restarting the agent, or providing additional feedback, but these strategies may either preserve incorrect states or discard useful progress. To address this issue, the authors propose agentrewind, a recovery framework that allows agents to return to previous execution checkpoints and continue from a more reliable state. The key idea is to jointly restore both the external environment state and the internal agent context. In addition to restoring workspace changes, agentrwind reconstruct previous interaction history and maintains rewind memory to record failed attempts, aiming to prevent the agent from repeating the same mistakes.
The authors also introduce MettleBench, a benchmark designed for evaluating long-horizon agent execution with multiple acceptance criteria. Experiments across different agent frameworks and tasks show that agentrewind improves success rates compared with continue, restart, and other recovery baselines. Ablation studies further analyze the contribution of environment rewind, context rewind, and rewind memory.
Critical strengths
The paper focuses on a practically important problem for long-horizon agents. As tasks become longer and involve more interactions, early mistakes can propagate and significantly reduce the probability of successful completion. The proposed recovery perspective provides a reasonable direction for improving agent robustness.
A notable strength is that the framework considers both environment state and agent context. Previous rollback mechanisms often focus only on external state restoration, while LLM agents also maintain internal assumptions through conversation history and previous tool interactions. Restoring both aspects is a meaningful engineering consideration.
The experimental evaluation is relatively comprehensive. The authors test across multiple agent frameworks and provide component-level ablations to understand the effect of different recovery elements. The introduction of a benchmark with multiple acceptance criteria also attempts to evaluate long-horizon execution beyond simple final success metrics.
Critical weaknesses
- The primary weakness is the limited methodological novelty. The central idea of saving checkpoints and rolling back execution states has been widely studied in computing systems. Although agentrewind extends this idea by additionally restoring LLM context and storing rewind memory, the paper does not sufficiently demonstrate why this represents a fundamentally new capability for LLM agents rather than a natural engineering adaptation of existing recovery mechanisms.
2..Another concern is that the framework assumes the agent can identify when the current trajectory should be abandoned. However, one of the most difficult problems in long-horizon agents is recognizing that a plan has become invalid or that progress has stalled. Without a reliable failure detection mechanism, the proposed recovery process may not be activated at the appropriate time.
-
The generality of the evaluation is limited. The benchmark mainly consists of software engineering tasks with controllable workspace states, which naturally favor rollback-based solutions. It remains unclear whether agentrewind can provide similar benefits in broader settings such as web agents, embodied agents, or environments involving irreversible external actions.
-
The paper does not fully isolate the unique contribution of rewind compared with stronger restart-based alternatives. Since restart methods can also preserve previous experience through summaries or memories, additional analysis is needed to understand when maintaining previous execution state provides advantages over simply restarting with better contextual information.
-
MettleBench's 640 acceptance criteria were LLM-written. The authors use deterministic gates and human task-level checks, but explicitly say they did not hand-annotate every criterion. Because the main success result depends on those criteria, a stratified criterion-level false-pass/false-fail audit is needed before treating its measured advantage as fully validated. This is a validity gap, not evidence that the current scores are wrong.
Questions for authors
The authors should address the concerns raised in the critical weaknesses, especially regarding the methodological novelty of agentrewind, the dependency on reliable failure detection, and the general applicability beyond controllable workspace environments.
Other: I am curious whether the authors have tried evaluating the method with more recent frontier models (e.g. GPT-6 or other latest LLMs). I am not suggesting that additional large-scale experiments should be conducted during the rebuttal period, as such evaluations may require substantial resources and time. I would mainly like to understand whether the authors have any preliminary observations regarding how the proposed method performs with newer model generation