acceptodds
Under review as a conference paper at ICLR 2027

HIERAMEM: Evidence-Anchored Hierarchical Memory for Self-Evolving LLM Agents

Abstract

Self-evolving LLM agents improve over time by accumulating and reusing past experience through memory. However, constructing memory from prior interactions entails a fundamental tension between preserving evidence and abstracting knowledge: complete trajectories retain rich behavioral evidence but introduce substantial noise, whereas abstract memories facilitate reuse but often obscure their evidential basis. To address this tension, we introduce HIERAMEM, an evidence-anchored hierarchical memory framework that bridges complete interaction trajectories and reusable knowledge through compact, traceable evidence anchors. During memory formation, HIERAMEM contrasts related trajectories with different task outcomes to identify behaviorally discriminative action–observation evidence and retain it as evidence anchors. These anchors constrain the abstraction of transferable decision knowledge, forming a three-level hierarchy comprising episodic evidence, evidence anchors, and reusable memory. As experience accumulates, this hierarchy evolves across episodes while keeping reusable knowledge grounded in its supporting anchors. At inference time, the same anchors support memory retrieval at two granularities, serving as global references at the task level and as local addresses at the decision level. Experiments across five benchmarks show that HIERAMEM improves overall agent performance over existing memory systems, demonstrating the effectiveness of evidence anchoring in connecting memory abstraction with reliable reuse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, self-evolving LLM agents, experiential memory, evidence anchors, hierarchical memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9FE7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HIERAMEM, a "three-layer" hierarchical memory system to improve self-evolving agents. The motivation of work is clear that task-specific episodes may contain noise that couldn't not generalize to other tasks. So HIERAMEM proposes an Evidence Anchor module that helps transofrm episode level knowledge to re-usable abstract knowledge. The authors present comprehensive empirical evidence that HIERAMEM can outperform previous Memory systems with some analysis showing more in-depth insights on how their system works.

However, the presentation is very unclear. I encourage the authors to significantly simplify and clarify the methods for the reader to easily follow. I think the author should just highlight the key differentiator in their work with examples or with better illustrations and keep other parts as straightforward as possible. For these reason, I think the paper requies another revision.

Critical strengths
  • As mentioned in the paper summary above, the motivation of the work is very clear.
  • The overall method is intuitive to address the issues from "prior" works.
  • Comprehensive experiments definitely add points.
Critical weaknesses

The biggest weakness of the paper is that the authors try to over-complicate the proposed method by throwing in too many math notations in the method section. This backfires. Compared with existing solutions (not just from academic papers, but also industry level products) to construct generalizible memory, the proposed method appears to be similar. This is acceptable if authors can clearly demonstrate the details and key differentiator of the method. However, the current narrative adds to the confusion. Here are a few concrete examples.

  • Eq(4): The choice of using goal is unclear here and the goal wasn't explain previously. This can be easiy illustrated with an example.
  • Eq(5): z and u are very abstract and z is defined with 4 different operations again, each of them are abract.
  • Eq(6) + Eq(8): P are not explained - though later I understand these are prompts. Also, (d, S, m) is defined by m and A, but A is defined with m again. The logic is convoluted. Simple example or illustration works much better.

Another weakness (which is acceptable) is that the tested benchmarks are verifiable with clear outcomes. However, agents nowadays are moving towards non-verifiable real-world tasks. So good to test if the proposed method can generalize there.

Questions for authors
  • Do you think the Evidence Anchor is another level or just a bridge between the specific knowledge to the more generalizible knowledge? If you think it is a level of memory, can downstream tasks retrieve this layer directly?
  • L172-173, what exactly is cold-start information and how do you derive memory from it alone?
Reviewer_BUYTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how LLM agents can preserve the interactive evidence necessary for judging the applicability of experiences, while simultaneously abstracting historical experiences. The authors propose HIERAMEM, a three-layer memory structure that organizes complete interaction trajectories, evidence anchors, and reusable rules. Memories are formed by contrasting successful and failed experiences, and the associations between rules and evidence are maintained during cross-task updates. During task execution, success anchors provide global references, while failure anchors support the retrieval of local guidance based on the current interaction state. The authors compare their approach against baselines across five benchmarks and three backbone models. Furthermore, the effectiveness of the method is validated through module ablation studies, parameter analysis, and token cost analysis.

Critical strengths

A valuable contribution of this paper is the introduction of compact, traceable evidence anchors as an intermediate representation connecting concrete experiences with abstract rules, and the consistent integration of this representation throughout memory formation, updating, and usage. The design, where success anchors serve as task-level references and failure anchors provide decision-level guidance, is both intuitive and consistent. The experiments cover five benchmarks and three backbone models, with the full method achieving the highest reported mean performance across all 15 combinations. Ablation studies on major modules also demonstrate performance degradation when components are removed, while token consumption remains at a similar magnitude to most memory baselines. These results provide positive evidence for the broad applicability and practical utility of the proposed method.

Critical weaknesses
  1. Lack of direct evaluation on retrieval hit rate and rule applicability. The paper emphasizes precise memory reuse based on current decision conditions, yet the execution phase primarily relies on semantic similarity to select anchors. Success anchors are selected based on task goals, while failure anchors are filtered using thresholds based on task goals and recent interactions. However, semantic similarity alone is insufficient to guarantee that historical rules are applicable to the current state, especially when task descriptions are similar but key preconditions differ. The paper does not report metrics such as retrieval hit rate, rule applicability rate, or false trigger rate, nor does it describe an independent applicability verification mechanism. While Table 2 demonstrates that the retrieval mechanism improves overall task performance, it does not prove that it reliably retrieves the experiences needed for current decisions; thus, the support for the claim of "precise and reliable reuse" remains limited.
  2. Lack of experiments isolating the contribution of the success–failure contrast. The success–failure contrast is a key design in memory formation. However, the "direct abstraction" variant in Figure 2 still uses the same success–failure trajectory pairs, meaning this experiment cannot independently verify the contribution of the contrast itself. Although the paper allows memory formation based solely on failure trajectories during cold start, it does not systematically compare three settings: success-only, failure-only, and success–failure contrast. Without controlled experiments where the amount of experience and generation budget are comparable, it is unclear how much additional benefit the success–failure pairing brings compared to unilateral experience.
  3. Some performance margins are small, insufficient to prove stable gains over competitive methods. In Table 1, the gaps between the proposed method and the strongest baselines are narrow in some cases. For instance, Qwen3-8B improves over ExpeL by only 0.25 percentage points on ALFWorld, and Qwen3-14B improves over the highest baseline by 0.67 percentage points on FEVER. The paper reports only the average of three runs, without providing inter-run variance, confidence intervals, or paired statistical analysis. Consequently, it is difficult to determine whether these slight leads exceed run-to-run fluctuations. While the results support the highest reported means, they are insufficient to confirm stable superiority over competing methods across all evaluation configurations.
  4. Insufficient disclosure of evaluation protocols and implementation details affects interpretability and reproducibility. The appendix does not clearly list the actual number of samples, data splits, task selection methods, and execution order used for each benchmark, nor does it specify which random factors were varied across the three independent runs. For methods that accumulate memory online, task order and the scope of historical experience directly impact subsequent task performance. Furthermore, HIERAMEM starts accumulating memory online from an empty state, whereas ExpeL uses four-fold validation and experience collection on training tasks; the correspondence of data splits, the visibility scope of experience, and the scope of cost statistics need further clarification. There are also gaps in implementation disclosure: the fixed text serialization template used for retrieval is not fully provided, the benchmark_specific_rules in Appendix D do not detail the specific content for each benchmark, and the cold-start prompt Pf′ is not listed separately and completely. These configurations affect similarity calculation, interpretation of interactive evidence, and initial memory formation, and should be provided in full to support reproducibility.
  5. Algorithm 1 omits the handling branch for an empty set of success anchors. Line 1 of the algorithm initializes memory as empty, and Line 4 immediately performs an argmax operation on the set of success anchors, which is undefined at that point. While Section 4.3 of the main text clarifies that no global reference is provided when there are no success anchors, the pseudocode should include the corresponding conditional check to ensure the algorithm description aligns with the actual cold-start process.
Questions for authors
  1. Is there a rule-applicability check during execution that is not described in the main text? Have the authors evaluated the proportion of retrieved rules that are actually applicable to the current decision, as well as the rate of false triggers? Can the system effectively distinguish cases that are semantically similar but differ in key preconditions?
  2. Under comparable amounts of experience and generation budget, what results are obtained by success-only, failure-only, and success–failure contrast settings? Given that the paper already includes a cold-start mechanism that forms memories from failure trajectories alone, how large is the performance gap between this mechanism—when extended to the full evaluation—and the default method?
  3. What are the results and ranges of variation across the three runs for the proposed method and the strongest baseline? For small margins such as 0.25 percentage points on ALFWorld and 0.67 percentage points on FEVER, is there evidence that the advantage is stably maintained across runs or task orders?
  4. What are the actual numbers of samples, data splits, task selection procedures, and execution orders used for each benchmark, and which factors were varied across the three runs? How does HIERAMEM’s online experience accumulation correspond to ExpeL’s four-fold validation? Which experiences are accessible to each method during evaluation, and which stages are included in the cost statistics?
1 comment

Thank you for the detailed response and additional experiments. The Direct SYM Retrieval comparison provides more direct evidence for the value of anchor-mediated retrieval, and the success-only, failure-only, and contrastive comparisons strengthen the evidence for memory formation. Together with the added variability statistics and protocol clarifications, these results make the contribution of the evidence-anchor layer more convincing. I will raise my rating to 3.

Reviewer_PL9ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the trade-off between preserving interaction evidence and abstracting reusable knowledge in memory for self-evolving LLM agents. It proposes HieraMem, a three-level memory (episodic evidence, evidence anchors, reusable memory) in which success–failure episode pairs are contrasted to localize decision-relevant anchors; these anchors constrain the abstraction of reusable guidance, are carried through ADD/REWRITE/NOOP consolidation while keeping anchor–memory links, and are reused at two granularities — a fixed task-level success reference and context-conditioned failure-anchor retrieval that addresses linked memories. Evaluation spans five benchmarks and three backbones against five experiential-memory baselines, with ablations on formation, evolution, and both utilization levels.

Critical strengths

Strength1: Evidence anchors are given a concrete dual role rather than being posited as a representation. The same anchors constrain abstraction during formation and serve as retrieval addresses at inference (Eqs. 12–13), and Table 2 tests exactly this: replacing either the task-level or decision-level anchor with the matched full trajectory, or removing it. That the full-trajectory variant can fall below providing no task-level reference at all is the most informative result in the paper, since it isolates selectivity of retained evidence rather than mere availability of past experience.

Strength2: The formation and evolution contributions are separated by controlled variants. Figure 2 replaces anchored formation with direct abstraction from the same contrasted pair, and replaces evolution with append-only accumulation, so the two mechanisms are not credited jointly. The append-only control is the right comparison because it holds the candidate-generation process fixed.

Critical weaknesses

Weakness1: Stability evidence does not match the granularity of the claims. The paper claims best performance in every backbone–benchmark setting (Sec. 5.2), but Table 1, Figure 2, and Table 2 report point estimates with no dispersion, and several margins over the strongest baseline are narrow. Results are averaged over three runs (Sec. 5.1), yet what differs across those runs is not stated, so the reader cannot judge whether the ranking or the ablation gaps are reproducible rather than artifacts of a particular execution.

Weakness2: No simple control isolates the anchor indirection from context-conditioned retrieval. Table 2 contrasts anchor-mediated retrieval against full trajectories and against no retrieval, but both alternatives change the retrieval unit and its length simultaneously. A compact memory could instead be retrieved by matching the current context directly against its own trigger-cue field (SYM in Eq. 5), with no anchor layer in the retrieval path. Without this control, the gains are consistent with "retrieve compact, context-matched guidance" rather than with anchoring specifically, which is the paper's central claim.

Questions for authors

Question1: What varies across the three runs, and what is the dispersion for Table 1 and the ablations? Even a few repetitions on two benchmarks with intervals would indicate whether the ablation gaps exceed run-to-run noise.

Question2: Could you add the control in Weakness2 — retrieving reusable memory by current-context similarity to its trigger-cue field, bypassing failure anchors, with formation and evolution unchanged? If anchored retrieval still wins, the indirection is doing work beyond compactness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.