- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focusses on the mismatch and effects of agent state and environment state in the case of an interruption as tested on AppWorld. The experimental design focusses on switching between keeping the agent and environment state in sync and not doing so and results show a fall when there is mismatch. Further trying this on SOTA models shows they can often discover and repair the missing state but doing so requires additional effort that ordinary success rates don't capture. Expanded to Cybench, the experiment demonstrates that even a perfectly restored agent memory can be insufficient for decision-making when important external state is missing or has changed.
Critical strengths
- The paper isolates the effects of incomplete restoration by keeping agent history and interpreter state fixed while varying the underlying application state.
- The Complete/Partial/Repair comparisons and categorization of unsuccessful or unqualified instances strengthen the reliability of the findings.
- The evaluation of SOTA models demonstrates they can frequently reconstruct missing application state but often require additional execution requests and sometimes declare completion despite missing required effects.
- Measuring recovery effort alongside task success is a useful contribution to agent evaluation.
Critical weaknesses
- The paper's central observation is already well established in recent agent-recovery research. Crab and AgentRewind address consistency between agent and environment checkpoints, while DART explicitly formalizes semantic recoverability and evaluates recovery on LangGraph. Safe to Resume? (2026) studies external-state mismatches and execution-continuity failures across existing agent frameworks.
- The omission of DART and Safe to Resume? from the related-work discussion further weakens the novelty positioning.
- The proposed decision-contract framework and Cybench examples largely formalize established principles of state consistency and partial observability without introducing a new recovery method.
- The primary experiments deliberately combine a later agent checkpoint with an earlier application database snapshot creating the inconsistency being investigated and making it synthetically likely grounds for the effect.
- Although the paper reports 270 adaptive episodes, these involve repeated evaluations of just nine selected affected instances, with 15 of the 17 Partial-restoration failures concentrated in a single payment-reminder task.
Questions for authors
- Without evaluation of naturally occurring interruptions, realistic checkpointing failures, or a broader distribution of recovery tasks, the work does not convincingly establish significance. Can we do that?
- How does the proposed restoration-completeness framework substantively advance beyond prior work as mentioned above?
4 comments
Authors' Response (follow up for Q1 and W5)
Thank you again for your careful review. One further study bears directly on Q1 and W5.
Q1. Without evaluation of naturally occurring interruptions, realistic checkpointing failures, or a broader distribution of recovery tasks, the work does not convincingly establish significance. Can we do that?
The same consequences appear in a complete, unmodified agent under real crashes. We ran the Hermes agent [1] with its native session recovery and its own memory. On six ToolSandbox tasks, we killed its gateway process 30 times, each after the application committed a change and before the agent recorded the result. Standard recovery then repeats ten contact creations and messages. Receipt-aware recovery reduces this to zero and cuts API calls from 82 to 60. In five direct-message cases, both runs receive the identical native score, 0.9253, although one sends every message twice.
W5. [...] with 15 of the 17 Partial-restoration failures concentrated in a single payment-reminder task.
These effects do not depend on the reminder task. The Hermes duplicates arise in contact creation and direct messages. In all four fault conditions of two CrewAI workflows, the run that repeats effects scores higher, for example 0.9315 versus 0.8679 with 10 versus 0 duplicates.
We have added these results to Section 5 and the appendix.
References
[1] Nous Research. Hermes Agent, 2026. https://github.com/NousResearch/hermes-agent
Thank you for the detailed response and the additional experiments. The container restart study addressed my concern about the deliberately mismatched snapshots. Both models recovered all eight tasks after storage loss but each needed three times as many requests as after a restart with durable storage. The additional failures across several non-reminder tasks also address my concern about the original results being concentrated in one task.
My main reservation is still novelty given the overlap with DART and 'Safe to Resume?'. The Hermes result helped clarify what this paper adds: in five cases, sending a message once and sending it twice received exactly the same score. I think that is a useful finding for people evaluating recovered agents. How often these failures occur in deployment remains open, but the revised paper makes a stronger empirical case. I’m raising my rating to 3 (weak acceptance).
Thank you very much for your thoughtful follow-up and for raising your rating. We are glad that the container restart study and the failures across non-reminder tasks addressed your concerns. We also truly appreciate your recognition of the Hermes finding. As you noted, sending a message once and sending it twice can receive exactly the same score, and we share your view that this matters for anyone evaluating recovered agents.
We will keep the distinction from DART and "Safe to Resume?" clear in the final version. Thank you again for your careful reading and constructive feedback throughout the discussion, which have made our paper stronger.
Dear Reviewer,
As the deadline for the discussion period approaches, we kindly request your feedback on our response. Your insights would be greatly appreciated.
Thank you for your time and consideration!