acceptodds
Under review as a conference paper at ICLR 2027

Closed-World Checkpoint: Asymmetric Recovery for Tool-Using LLM Agents

Abstract

Agent recovery has two temporal directions: internal task state rewinds, and knowledge of committed external actions persists. A stale privilege or completion marker can misdirect the resumed task; erased effect history can trigger duplicate execution. We identify an asymmetric recovery rule that coordinates the agent's internal basis for action with its committed history. Closed-World Checkpoint (CWC) realizes this rule through a registered closure of workspace, trajectory, side stores, and effect receipts, coordinated at a single resume boundary. Crossed member ablations expose the opposing failures: omitted side-state recovery permits unauthorized action, whereas lost receipts permit duplicate effects. In the primary authorization stress test, qwen-plus issues deletion calls in all 10 workspace-and-trajectory recovery trials; CWC preserves the file in all 10 paired trials. Longer LLM continuations and LangGraph integration retain this dependency. Browser-use restart experiments resolve committed orders through either a retained receipt or stable operation identity; an effect-before-receipt experiment uses outbox and sink idempotency to resolve the earlier crash window. Together, these results make the next tool call the organizing point of recovery: its authority and obligations must match the restored task, and its external actions must remain connected to what has already committed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent rollback, checkpoint restore, tool-using LLM agents, side-store closure, egress honesty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ErfvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how tool-using LLM agents can continue safely after restoring an earlier checkpoint. Incomplete recovery can leave permissions and task records inconsistent with restored files and conversation, or cause already completed external actions, such as sending an email, to be repeated.

The main contribution is an explicit recovery rule: internal task state returns to the checkpoint, while records of completed external actions are preserved. The proposed Closed-World Checkpoint implements this rule by restoring the agent's registered internal task state while retaining effect receipts. It coordinates these steps before allowing tool use to resume.

Experiments across different execution settings show that CWC helps prevent unauthorized actions, incorrect completion reports, and repeated external operations in the tested settings. However, if an external action succeeds before its completion record is saved, CWC alone may still repeat it, so additional mechanisms are needed to make retries safe.

Critical strengths

The ablation experiments clearly show the different roles of side-store recovery and receipt preservation. Removing side-store recovery brings back unauthorized deletion, while removing receipts causes repeated external operations; in each case, protection against the other failure remains intact. Section 4.4 also reports a crash scenario in which CWC alone fails, helping clarify the method's limitations. The ablations make the proposed mechanism more convincing.

Critical weaknesses

CWC's added research contribution is not sufficiently established. Section 4.6 and Appendix G report identical outcomes for CWC and FS when recovery state and protective mechanisms are matched. The origin and construction of FS, the specific hypothesis tested by this comparison, and how its results support CWC's contribution need clearer explanation. This leaves unclear how much the paper advances existing recovery ideas through its recovery rule or implementation. The paper does not yet provide a convincing case for CWC’s added research value.

Questions for authors

1.Section 2.1 defines GtG_t as the agent's effective constraints/obligations derived from WtW_t and MtM_t; how is the "do not send" constraint represented in GtG_t if the user instruction "draft the email but do not send it" exists only in τt\tau_t? Does this assume that all relevant constraints/obligations in τt\tau_t are recorded in WtW_t or MtM_t, or should GtG_t also depend on τt\tau_t?

2.Please clarify the origin and recovery procedure of FS and how it differs from CWC. Given their identical outcomes under matched recovery and continuation settings, what does this comparison establish about CWC's added research contribution?

Reviewer_vsebOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a closed world checkpoint (CWC) abstraction, which allows rolling back AI agents that call tools and store state. The key idea is to include in the rollback anything outside the agent that it could have touched, e.g., files, memory, to-do flags, permission grants, etc. thereby avoiding problems such as stale permissions or false completions after restore. The authors show that their approch works in scripted test scenarios and produces small integration demos with LangGraph, browser use, Gmail, and Stripe.

Critical strengths

(S1) Rollback and restoring conversation and workspace in a safe way is a very real and under-appreciated problem for agentic systems.

(S2) The paper is honest about its negative results and limitations.

(S3) The failure taxonomy is clear, useful, and easy to remember for people building agent-take systems.

Critical weaknesses

(W1) The paper's contribution (CWC) is defining requirements for resuming agents safely, which is in itself quite obvious. What would have been interesting is to show that existing production systems exhibit this problem and demonstrate that CWC remove them in practice. As it is, the novelty boils down to requiring that you list all pieces of the state that can influence checkpointing and that you restore them all, which is a basic precondition of a consistent checkpoint.

(W2) The experiments fail to demonstrate that existing systems suffer from these failures in the real world and that using CWC removes them. The LLM runs are largely unconvincing in that regard since they use a custom harness and essentially plant the stale grant in a side store that the naive baseline skips. The experiments using real frameworks (specifically LangGraph and browser use) exhibit the same experimental design flaw: the authors deliberately place state outside the state LangGraph restores. The browser use case restarts the agent without checkpointing, which inevitably results in the non-idempotent retry. These outcomes are expected from the experimental design and are, therefore, unsurprising. Overall, I would have liked to see the mechanism evaluated in a less contrived fashion.

(W3) The CWC approach seems to break in the event that a restarted agent does not know about past effects. The main text does not explain how the model is told about a receipt. Scanning the appendix, it seems that the authors rely on the tool call recognizing a duplicate call caused by a rollback to be the same operation by matching, for example, on an identifier (e.g., message ID, order key). However, this assumes that the continuation from the checkpoint is deterministic, which cannot be true unless the steps are fixed in advance. Other systems in this space typically go to great lengths to ensure that idempotency keys survive checkpoint rollback no matter what. I would ask the authors should dedicate a section to this important issue in the paper, and, if they choose to stick to the current design, it should be noted as a limitation.

(W4) Beyond idempotency keys, the fault tolerance story still does not seem completely worked out. For example, restore is not atomic, which means that a crash partway through would leave the workspace, conversation, stores, and ledgers inconsistent. Similarly, unless I misunderstood, the wrapper writes a receipt with every effect, which essentially fails to cover the crash window between an effect and its receipt (the harder case is not covered). Finally, the paper does not explain how CWC deals with network stores, browser cookies, shared storage, in-process state, etc.

(Minor) The fit is potentially questionable for ICLR given that the paper introduces a systems recovery mechanism and treats LLMs as a blackbox. My score does not reflect this, but in the event the paper is not accepted at ICLR, I would advise the authors to consider systems or database venues as a potential resubmission targets.

Questions for authors

Please see details of the weaknesses section.

(Q1) Please clarify if I missed anything experimental or claimed novelty relative to W1 and W2.

(Q2) How would you deal with idempotency keys properly?

(Q3) Please fill the gaps for fault tolerance (see W4)

Reviewer_x66YOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that restoring a tool-using LLM agent after a checkpoint requires asymmetric state handling in two temporal directions. Internal decision state (workspace WW, trajectory τ\tau, and registered side stores MregM^{\text{reg}} such as grants and obligation markers) must rewind to the checkpoint. Knowledge of committed external effects (receipt ledger EmetaE^{\text{meta}}) must persist and merge with post-checkpoint receipts. The proposed Closed-World Checkpoint (CWC) uses operator registration of side stores (A1) assisted by a scanner ("N3"), wrapper-enforced egress receipt logging (A2), and a resume gate that pauses tool calls until restoration is complete. Three failure families are defined: R1 duplicate egress, R2 stale privilege leading to unauthorized deletion, and R3 false completion. Evaluation covers deterministic tool loops, crossed ablations, HTTP/SMTP egress, crash windows handled by an outbox with idempotency, hosted/self-hosted LLM continuations (qwen-plus, deepseek-chat, glm-5.3, Mistral-7B, Qwen2.5-14B), a LangGraph integration, browser-use restarts, and a 144-execution continuation matrix.

Critical strengths
  1. Timely Problem Framing: The paper clearly articulates a real problem in agent reliability: frameworks checkpoint conversation histories and graph nodes, but authorization caches, memory files, and completion markers often reside outside that state, leading to stale privilege bugs upon restore.
  2. Clean Causal Isolation via Crossed Ablations: The crossed member ablation (Table 3) effectively isolates component dependencies: removing side-store recovery reopens R2 while keeping R1 closed, and removing receipt preservation reopens R1 while keeping R2 closed.
  3. Honest Boundary Attribution: The paper clearly distinguishes what CWC resolves from what it does not, explicitly showing that the effect-before-receipt crash window defeats CWC alone and crediting the transactional outbox plus sink idempotency keys for closing it.
  4. Broad Practical Coverage: The engineering coverage spans process boundaries (HTTP/SMTP), Stripe test mode, LangGraph 1.2.11 interrupts, and browser-use 0.13.10 restarts, accompanied by low runtime overhead (+1294 bytes per checkpoint, median restore time 1.54 ms).
Critical weaknesses
  1. Main Results Hold by Construction: In every R2/R3 scenario, side-store grants or completion markers are placed in stores that C0 does not restore and CWC does (Section 4.1). Once the executor or LLM conditions actions on that store, C0 failing and CWC succeeding is guaranteed by setup. Furthermore, manual cleanup (B3) matches CWC across all failure tables (Tables 13–14, 18–20, 24), and the matched complete-state baseline ("FS", Section 4.6) matches CWC identically across all 20 PayRemind runs and 144 continuation executions. The experiments confirm implementation correctness but do not demonstrate superior efficacy over complete snapshots and standard idempotency.
  2. LLM Evaluation Protocol Flaws and Forced Actions: Section 4.5 reveals that the R2 stress protocol uses grant-conditioned prompt retries followed by a deterministic fallback executor that executed deletions for deepseek-chat seeds 42 and 45. Table 6 counts these as 5/5 model breaks, meaning the deletion was not model-issued in 40% of DeepSeek trials.
  3. Non-Independent Replicates and Invalid Statistical Bounds: Hosted model runs used temperature zero with identical state and prompts across seeds 42–51 (Section 4.5). These runs are deterministic replicates rather than independent Bernoulli trials, making the reported Wilson 95% confidence intervals (e.g., lower bound 0.722 for 10/10) statistically invalid. Sample sizes elsewhere are minimal: R1 and R3 use 1 seed per model (Tables 7–8), longer continuations use 1 seed per model (Table 9), browser-use uses 1 trial per configuration (Table 28), and external APIs are tested once (Section 4.4).
  4. Limited Novelty Over Existing Systems: The core mechanism combines standard rollback-recovery, transactional outboxes, deduplication receipts, and idempotency keys (Elnozahy et al., 2002; Helland, 2012). The main agent-specific addition is registering side stores, which relies on operator declaration (A1). The assistive scanner N3 yields 427 false positives out of 506 hits and a 0.15 planted false-negative rate with no denominator provided (Appendix H.1). Comparisons against close systems like ACRFence or Atomix remain purely conceptual.
  5. Assumed "Closed-World" Completeness and Out-of-Scope Risks: Completeness depends entirely on assumption A1. If an operator misses an action-relevant database row, session, or environment variable, all guarantees vanish. Furthermore, receipts are trusted against tampering, credential revocation is unhandled, and settlement depends on external service idempotency support (Section 6).
Questions for authors
  1. What are the exact prompt retries, retry limits, and trigger conditions used in the Section 4.5 LLM protocol? What are the C0 deletion rates when no prompt retries and no deterministic fallback executors are used?
  2. At temperature zero, did model outputs or trajectories differ across seeds 42–51? If not, will you report n as the number of distinct trajectories and remove the Wilson intervals?
  3. How does CWC compare empirically in runtime overhead and failure prevention against (a) storing grants inside LangGraph state channels directly, and (b) ACRFence or Temporal runtimes under the same R1–R3 faults?
  4. What was the total denominator of planted stores used to calculate N3's 0.15 false-negative rate in Appendix H.1?
  5. How do you reconcile Section 4.5's claim that CWC "rewrites the missing deliverable" with Table 22's entry llm_skip_dirty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.