- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies memory for recovering an interrupted agent task. RecoverMemBench distinguishes resumption, where the environment retains completed work, from rollback, where the environment is restored to an earlier state but memory may still describe reverted actions. It constructs resumption cases and rollback cases from SWE-bench tasks that the original agent solved successfully in three runs. The proposed RecoverMem retrieves step-indexed structured state snapshots and retains reflections from failed continuations.
Critical strengths
Pros
-
The problem formulation captures a practical memory requirement. Resumption and rollback expose different consistency conditions. Remembered progress may remain valid in one setting and become invalid in the other. This distinction extends memory evaluation beyond recalling historical facts or transferring experience to a new task.
-
The benchmark makes a meaningful effort to separate recovery from basic task-solving ability. Selecting tasks solved successfully three times reduces ambiguity about whether a failure merely reflects an unsolvable task.
-
The reference method is simple and supported by informative ablations. Step-indexed state retrieval directly addresses memory–environment mismatch, while failure reflection retains useful lessons from an unsuccessful continuation.
Critical weaknesses
Cons
-
The evaluation is narrow relative to the deployment claims. The 416 cases derive from only 104 independent tasks, with original trajectories averaging approximately 13.89 steps. The evaluation primarily uses one executor model, injected rollback failures, and a known recovery step.
-
The comparison with version-aware baselines materially qualifies the method advantage.
Questions for authors
Questions
-
How would RecoverMem identify valid state when there is no reliable global rollback step?
-
How can the system retain knowledge from a failed suffix without reintroducing invalid progress?