acceptodds
Under review as a conference paper at ICLR 2027

Yet another agent memory system? Hippo: training-free self-evolution via hippocampal replay

11 views

Abstract

Motivated by recursive self-improvement (RSI), we study how LLM agents can learn through external memory without updating model parameters. Extracting useful experience from a single attempt is difficult: flawed reasoning can produce success, while poor execution can obscure a sound strategy. Preserving its value over time is also difficult: our accumulation baseline without consolidation initially improves but eventually performs below a no-memory baseline. Our key idea is to compare attempts at the same task to identify reusable decisions and refine when they apply. We introduce Hippo, a memory framework inspired by selective hippocampal encoding and replay that learns from these comparisons when shortcomings are diagnosed. It compares the resulting experience with existing memory to merge overlaps and resolve conflicts through explicit applicability conditions. This allows prior guidance to be revised without storing every lesson as a separate entry. Experiments on SWE-bench Verified, WebArena, and Mind2Web show performance gains in all three settings and substantially fewer retained experiences than the accumulation baseline. The gains extend across the evaluated base models of varying capabilities, indicating that memory-based adaptation remains useful even for highly capable models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving LLM agents, Agent memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZRgTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Hippo improves agents with frozen model parameters by comparing multiple trajectories, reflecting on failures and fortuitous successes, and updating reusable textual memories. On WebArena and SWE-bench, memories are updated after all attempts on a task and used by subsequent tasks. Experiments also cover Mind2Web and a longer learning stream.

Critical strengths
  • Tables 1–2 show consistent improvements across four backbones. With GPT-5.6 Sol, success increases from 64.2% to 72.9% on WebArena and from 81.5% to 88.5% on SWE-bench Verified.
  • The long-horizon experiment includes both revisited and unseen tasks, with a useful append-only control: full Hippo achieves 86%/82% versus 72%/66% after 400 learning tasks (Table 11).
  • The appendix provides concrete memory examples, failure cases, and a human audit of the judge.
Critical weaknesses

My main concerns are the limited methodological novelty, gaps in the experimental evaluation, and the lack of code provided to reproduce the reported results.

  1. Novelty and baseline coverage remain insufficiently established. Training-Free GRPO (preprint, 2025) already derives semantic advantages from grouped rollouts and updates textual experience; ExpeL (AAAI 2024) compares successes and failures and revises insights. Mem0 (ECAI 2025) also updates and deletes existing memories, albeit in a different application setting. The external memory baselines are limited to AWM and ReasoningBank, leaving out representative alternatives such as ACE (ICLR 2026) for incremental maintenance and MemRL (NeurIPS 2026) for utility-based retrieval. Surprise handling and learning from all-failure batches are plausible distinctions, but the judge ablation removes both filtering and diagnostic text, preventing clear attribution. The paper needs a sharper account of its incremental contribution and a focused comparison under matched conditions.

  2. The Best-of-N and efficiency questions are not directly answered. Table 3 reports only No Memory's mean per-attempt success (64.3%) and any-of-eight success (73.6%); Hippo has no corresponding any-of-eight or verifier-selected result. These cannot be compared directly with Hippo's main-table mean success, and the domain aggregation also differs. Moreover, the reported cost increase from $1.93 to $2.17 per task (+12.4%) compares two methods that both use eight rollouts. It measures overhead beyond repeated sampling, not the full cost relative to single execution. The cost proxy states that assessment is included, but the call breakdown does not clearly locate trajectory-judge calls. Thus, superiority over direct retries at equal total budget remains unestablished.

  3. The evaluation supports online adaptation more directly than independent generalization. The main experiments update memory throughout the evaluated stream. WebArena uses reference-assisted memory construction, including for the memory baselines; Mind2Web uses gold-derived candidates and carries memory across splits. These protocols need to be distinguished from evaluation with frozen memory on independent tasks. Section 4.2 does include a held-out set, but the task composition and learning/test relationship are insufficiently specified. The main experiments hold task order fixed across seeds, and the long-horizon results lack an order-robustness check.

  4. Reproducibility is difficult to assess. No separate supplementary materials were submitted. Although the main PDF includes appendices, some implementation and evaluation details remain insufficiently specified, making it difficult to establish whether the reported results can be independently reproduced.

  5. Minor comments. Clarify which concrete design choices follow from the hippocampal replay analogy. “Misleading or overly trajectory-specific lessons” would be clearer than the undefined “overfitting a lesson”. Fix “basedline” and the duplicated “(i)” in Section 3.3. The improvement reported in Section 4.3 is 0.9 percentage points, not a 0.9% relative increase.

Questions for authors
  1. Can you compare against one close trajectory-comparison method (Training-Free GRPO or ExpeL) and one maintenance/retrieval alternative (ACE or Mem0 or MemRL), adapting them to the same feedback and task stream?
  2. Can Hippo and No Memory be evaluated with the same success/selection criterion and total budget, including judge and memory costs, across different rollout counts?
  3. Could you provide a little more detail on the Mind2Web evaluation setup and how tasks are selected and divided in the long-horizon experiment? This would help readers better understand the scope of the reported generalization results.
Reviewer_85UbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies memory-based self-improvement for agents without updating model parameters. The paper argues that a single trajectory can produce misleading lessons and that continued memory accumulation can eventually hurt performance. To address these issues, it proposes Hippo, which compares multiple attempts at the same task, extracts reflections from failures or successes with flawed reasoning, and consolidates the resulting experience into reusable memories with explicit applicability conditions. Experiments on SWE-bench Verified, WebArena, and Mind2Web report improvements across several backbones. A longer task stream also shows that Hippo maintains better performance with fewer memory entries than the evaluated accumulation-based alternatives.

Critical strengths
  • The framework directly addresses two plausible failure modes of agent memory: learning an incorrect lesson from a successful attempt and accumulating conflicting memories. Comparing attempts and maintaining explicit applicability conditions provide a coherent approach to these problems.
  • The empirical evidence is relatively substantial, with three task settings, four backbones in the main WebArena and SWE-bench comparisons, three-run results, and a longer task stream. The human audit of the surprise judge and reporting of memory-operation costs further support the practical value of the reported gains.
Critical weaknesses
  • The evaluation does not fully establish the claimed generalization. Appendix C uses fixed task-ID orders for SWE-bench and WebArena, while Mind2Web uses gold action histories, gold-derived candidate sets, and persistent memory across splits. WebArena also supplies the judge with official outcomes and reference answers. The baselines share that reference access, but the results still depend on a feedback regime that may not be available in deployment. The existing 50-task unseen evaluation does not fully resolve these issues for the main comparisons.
  • The ablations do not cleanly identify why the method works. Removing the surprise judge also removes textual diagnoses, while the repeated-rewrite baseline can already express replacement through deletion and addition. Its 54.57% result versus Hippo’s 72.90% in Table 5 therefore needs a more controlled explanation. This limits confidence in the specific mechanisms even though the overall empirical gains support acceptance.
Questions for authors

Please refer to my weakness part.

Reviewer_8gdYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hippo, a new memory framework for LLM agents that aims to identify reusable decisions and refine their applicability conditions. For each reasoning task, Hippo performs multiple independent attempts under the same conditions, identifies valuable information across the resulting trajectories, and distills reusable experiences to update its memory. Experiments on multiple datasets demonstrate consistent performance improvements comparing with baseline models.

Critical strengths
  1. The paper is well motivated, and memory construction is an important direction for improving LLM agents.

  2. The proposed method and evaluation protocol are clearly presented, and the paper includes extensive experiments to validate the proposed approach.

Critical weaknesses
  1. The proposed method requires multiple repeated executions of each task to accumulate experience, increasing both the time and computational cost of experience collection. Although the authors state that the same number of attempts is used for the baselines, the results are averaged without fully exploiting these trajectories for experience accumulation. In effect, some baselines use only a single trajectory to construct experience.

  2. The authors conduct experiments to verify the effectiveness of dynamically revised and expanded experiences. However, could this update strategy become ineffective in larger-scale applications? Further analysis and discussion would be helpful.

Questions for authors
  1. Compared with the results reported in the ReasoningBank paper, the performance reported here is lower on the WebArena dataset despite using newer LLMs. What might be the causes?

  2. The ablation results in Table 5 suggest that most of the performance improvement comes from utilizing multiple rollouts. Could the authors provide additional analyses or experiments to better demonstrate the benefits of memory revision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.