acceptodds
Under review as a conference paper at ICLR 2027

Remembering Is Not Recovering: Benchmarking Agent Memory for Task Recovery

Abstract

In real-world deployments, *large language model* (LLM) agents may encounter unexpected interruptions, such as network outages or machine failures, making reliable task recovery essential. However, existing agent memory benchmarks focus on using past experience to improve future tasks, rather than recovering the state of an interrupted task and resuming it without repeating completed work or skipping necessary steps. To address this gap, we introduce *RecoverMemBench*, which evaluates recovery rates and additional costs relative to uninterrupted runs under two settings: *task resumption*, which preserves progress and tests task state reconstruction, and *task rollback*, which restores an earlier environment state and tests whether memory identifies lost progress while retaining reusable information. Across 11 existing memory systems, average recovery rates remain below 50% in both settings, with over 150% additional steps in the rollback setting, highlighting substantial reliability and efficiency gaps. Our analysis reveals two key limitations: (1) semantic retrieval struggles to identify the information needed for recovery among similar execution records of the same task; and (2) inadequate memory state management causes agents to mistake work undone by rollback for completed work. Motivated by these, we develop a memory system that improves recovery rates by over 35% in both settings while reducing average additional cost by more than half. Overall, our work provides a foundation for evaluating and improving agent memory for reliable and efficient task recovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_idAbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory for recovering an interrupted agent task. RecoverMemBench distinguishes resumption, where the environment retains completed work, from rollback, where the environment is restored to an earlier state but memory may still describe reverted actions. It constructs resumption cases and rollback cases from SWE-bench tasks that the original agent solved successfully in three runs. The proposed RecoverMem retrieves step-indexed structured state snapshots and retains reflections from failed continuations.

Critical strengths

Pros

  1. The problem formulation captures a practical memory requirement. Resumption and rollback expose different consistency conditions. Remembered progress may remain valid in one setting and become invalid in the other. This distinction extends memory evaluation beyond recalling historical facts or transferring experience to a new task.

  2. The benchmark makes a meaningful effort to separate recovery from basic task-solving ability. Selecting tasks solved successfully three times reduces ambiguity about whether a failure merely reflects an unsolvable task.

  3. The reference method is simple and supported by informative ablations. Step-indexed state retrieval directly addresses memory–environment mismatch, while failure reflection retains useful lessons from an unsuccessful continuation.

Critical weaknesses

Cons

  1. The evaluation is narrow relative to the deployment claims. The 416 cases derive from only 104 independent tasks, with original trajectories averaging approximately 13.89 steps. The evaluation primarily uses one executor model, injected rollback failures, and a known recovery step.

  2. The comparison with version-aware baselines materially qualifies the method advantage.

Questions for authors

Questions

  1. How would RecoverMem identify valid state when there is no reliable global rollback step?

  2. How can the system retain knowledge from a failed suffix without reintroducing invalid progress?

Reviewer_yXDZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RecoverMemBench to evaluate how agent memory supports task resumption after interruption and rollback recovery after failure. It constructs 416 recovery cases from 104 SWE-bench tasks and compares 11 memory systems using recovery success and additional execution steps. The authors analyze inaccurate state retrieval and mismatches between memory and the restored environment, then propose RecoverMem, which stores and retrieves structured task states by execution step and incorporates failure reflection. Evaluation with a GPT-4o executor includes recovery experiments, state-based QA, and component ablations. RecoverMem achieves an average resumption recovery rate of 68.90% and a rollback recovery rate of 43.27%.

Critical strengths

S1: The paper explores an evaluation problem in agent memory and reports some findings.

RecoverMemBench distinguishes task resumption from task rollback and examines whether memory helps agents restore valid task state and continue execution. Selecting tasks that the agent can repeatedly solve without interruption helps reduce the confounding effect of basic task-solving ability (Section 2.3). Under the evaluated setting, the results suggest that retaining historical information does not necessarily support recovery: semantic retrieval may miss state information needed at the recovery point, while memory after rollback may describe undone work as completed (Section 3.2). These observations warrant further investigation.

Critical weaknesses

W1. The added value over straightforward recovery approaches is not sufficiently established.

The introduction argues that full interaction history is limited by context length and noise, but the paper does not directly evaluate this alternative or report the original history lengths in tokens. Since the recovery point (k) is known, rollback could use the history up to (k), or the full history with subsequent actions explicitly marked as undone. These comparisons are important for determining whether the proposed memory mechanism provides a substantive benefit.

The snapshot comparison also requires further analysis. In Appendix C.1, Table 4, snapshot-adapted A-MEM matches RecoverMem’s rollback recovery rate of 43.27% and incurs fewer additional steps: 70.56% versus 89.35%. RecoverMem does use less storage, but an advantage in rollback success or execution overhead over snapshot-adapted A-MEM is not evident.

W2. It is unclear how the gains depend on the underlying model.

Task recovery is evaluated with GPT-4o as the executor and GPT-4o-mini for memory construction (Section 3). It therefore remains unclear how much additional value RecoverMem provides when the executor is stronger or differs in its ability to use long histories. Comparisons across capability levels and within-family model sizes would help establish whether the gains are robust or mainly specific to the current model configuration.

Questions for authors

Q1: Additional evidence on full-history recovery.

Could you directly restore the complete interaction history for task resumption, and test both history up to (k) and full history with undone actions explicitly marked for rollback? Please report the history-length distribution and the trade-off between recovery success and cost, including results on cases where the complete history fits within the context window.

Q2: Additional evidence on model dependence.

Could you evaluate a small set of executors with different capabilities while keeping the memory construction model fixed—for example, different sizes within one Qwen family, together with a stronger Qwen, DeepSeek, or other model? The key comparison is RecoverMem’s gain over full-history and snapshot baselines under each executor, with that model’s uninterrupted task performance also reported.

Q3: Clarification of the rollback advantage.

Given Table 4, is RecoverMem’s advantage over the stronger snapshot-adapted baselines primarily lower storage consumption? Please clarify the rollback performance and cost claims accordingly, distinguishing storage and execution steps from total cost. Appendix C.2 already notes that the reported measurements do not establish total cost.

Reviewer_KFmiOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RecoverMemBench, built on SWE-bench, to evaluate whether agent memory systems help a fresh agent recover an interrupted task under two settings, task resumption and task rollback. It evaluates 11 existing memory systems and proposes RecoverMem, which stores a structured task-state record at every step and retrieves it by step index.

Critical strengths
  1. I think task recovery is a practically relevant aspect of agent reliability in the agent memory research filed.
  2. The benchmark covers both resumption and rollback, and evaluates a broad set of existing memory systems.
Critical weaknesses
  1. The problem setting is not well motivated: Practical systems persist session logs and checkpoint the environment and memory together; providing the full previous trajectory, which fits in context here, is not compared.
  2. The rollback mismatch exists only because the environment is rolled back but the memory is not; RecoverMem's step-indexed retrieval then amounts to a checkpoint.
  3. The benchmark does not support the claims: 104 tasks that GPT-4o solves reliably, with a median of 11 steps, one agent, one backbone, and one run per setting, far from the hundreds of actions in the motivation. Interruption points are set from the known length of successful runs, and rollback errors are injected by an LLM rather than made by the agent.
  4. The evaluation of RecoverMem is not convincing: It receives the git diff (Figure 7) and the rollback step index, which the baselines do not; with step-wise snapshots, baselines reach similar recovery rates (Appendix C.1).
  5. It is tailored to SWE tasks and evaluated only on the authors' own benchmark, not on existing memory benchmarks.
  6. The failure analysis is weakly supported: the "full memory bank" test uses the evaluation question as the retrieval query, and the questions match RecoverMem's record fields. The link between answer correctness and recovery success is not shown.
Questions for authors
  1. Without memory, resuming at 25% is close to restarting from scratch, yet the recovery rate is only about 22%, although the agent solves all these tasks in three runs. What explains this, and what prompt does the recovered agent receive?
  2. Additional steps include failed runs and exclude the cost of memory construction and reflection. The "over 35%" gain in the abstract is relative to the average baseline, not the strongest one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.