acceptodds
Withdrawn from ICLR 2027

H-Mem: A Novel Memory Mechanism for Evolving and Retrieving Agent Memory via A Hybrid Structure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory data are ubiquitous in Large Language Model (LLM)-based agents (e.g., OpenClaw and Manus). A few recent works have attempted to exploit agents' memory for improving their performance on the question-answering (QA) task, but they lack a principled mechanism for effectively modeling how memory data evolves over time and retrieving memory data effectively, leading to poor performance in memory utilization. To fill this gap, we present H-Mem, a novel memory mechanism via a hybrid structure that can not only effectively model the evolution of agent memory over a long period of time, but also provide an efficient memory retrieval approach. Particularly, H-Mem builds a temporal and semantic tree structure that allows the short-term memory data to evolve progressively into long-term memory data, where the latter provides summarized information about the former, while simultaneously constructing a knowledge graph to capture the relationships between entities in memory. Moreover, it offers an effective memory retrieval approach by exploiting the hybrid structure of the tree and graph structures. Extensive experiments on three agent memory benchmarks show that H-Mem achieves state-of-the-art performance on the QA task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Agents, Agent Memory, Retrieval-Augmented Generation, Long-Term Memory, Knowledge Graphs

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_P7gGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H-MEM, a long-term memory system for LLM agents that combines a temporal-semantic tree with an entity-centric knowledge graph. The tree progressively consolidates fine-grained events into longer-term memories over different temporal granularities, while the graph captures relations between entities across memory fragments. At retrieval time, an LLM-based planner decomposes queries, selects memory scope and coverage, traverses the graph and tree, and optionally issues a second retrieval query when information is missing.

H-MEM is evaluated on LoCoMo, LongMemEval-S, and REALTALK against several existing memory systems. The reported results show strong improvements in downstream QA accuracy and F1, together with ablations and indexing/retrieval cost analyses.

Critical strengths
  1. The system is fairly complete and practically motivated. The temporal-semantic tree and entity graph provide complementary views of the memory, and the provenance-preserving design is useful for maintaining access to the original evidence.

  2. The empirical evaluation is reasonably broad. H-MEM is tested on three long-term memory benchmarks, includes both simulated and human conversational data, and reports not only QA quality but also indexing/retrieval cost and component ablations. The improvements over the evaluated baselines are substantial on several settings.

Critical weaknesses
  1. The novelty and empirical positioning are not sufficiently convincing relative to the rapidly evolving agent-memory literature.

The current comparison includes Mem0, MemTree, MemoryOS, MemOS, Zep, and EverMemOS, but omits several recent methods that are highly relevant to the paper's central claims. In particular, recent work such as Memory-R1, MAGMA, APEX-MEM, and Agentic Memory already studies learned memory management, temporal/evolving memory, graph-based memory, or agentic retrieval, with several of them evaluated on LoCoMo and LongMemEval.

This matters because the paper claims state-of-the-art performance and presents temporal evolution, structured memory, and agentic retrieval as its main contributions. By 2026, these ingredients are no longer individually new, and the paper needs a much clearer comparison against contemporary methods to establish what the hybrid tree–graph design contributes beyond recent alternatives.

  1. The central claim of modeling memory evolution is only evaluated indirectly through downstream QA.

H-MEM explicitly claims to model the evolution from short-term to long-term memory, including stable, evolving, and conflicting information. However, the evaluation mainly measures final QA F1 and LLM-Judge Accuracy. These results show that the full retrieval system answers questions well, but they do not directly establish that the proposed consolidation process models memory evolution correctly.

For example, I did not find direct evaluation of whether evolving preferences are updated correctly, whether conflicts are preserved faithfully, whether consolidated memories remain supported by their source events, or how much information is lost at higher levels of the tree. This distinction is important because the full system also introduces query decomposition, memory-scope prediction, graph expansion, reranking, entity profiles, and second-pass retrieval. Thus, improved QA performance could partly come from a stronger retrieval pipeline rather than better modeling of memory evolution itself.

Questions for authors
  1. Could the authors clarify why recent methods such as Memory-R1, MAGMA, APEX-MEM, and Agentic Memory are not included in the comparison? Several of these methods are evaluated on LoCoMo and/or LongMemEval and appear directly relevant to H-MEM's claims about evolving, structured, and agentic memory.

  2. Do the authors have any direct evaluation of the memory-evolution mechanism itself, beyond downstream QA? For example, accuracy of preference/status updates, conflict preservation, consolidation faithfulness, or retrieval of outdated versus current information would materially affect my assessment of the central claim.

  3. In the "w/o tree" and "w/o graph" ablations, are the query planner, embedding/reranking stack, retrieval budget, and second-pass retrieval kept identical? I would like to understand whether these results cleanly isolate the contribution of the index structure.

  4. The paper states that fragment timestamps correspond to ingestion time rather than the time of the event described in the conversation. How are cases handled where event time and ingestion time differ substantially, and which notion of time is used for tree placement and temporal relevance scoring?

Reviewer_3rvsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose H-Mem, a memory architecture for question answering over dialogue histories. H-Mem combines a tree structure with a knowledge graph. Retrieval uses both structures and accounts for the temporal organization of events in the tree. The authors evaluate the method on popular long-term memory benchmarks: LoCoMo, LongMemEval-S, and REALTALK. Overall, H-Mem achieves better performance than the baselines.

Critical strengths
  1. Given the complexity of the architecture, it is described reasonably clearly.

  2. The experimental evaluation is solid. The authors compare against relevant baselines and pay attention to fair comparisons (e.g., by using the same embedding models for RAG-based methods, lines 283–285). They also conduct a meaningful ablation study, although it could be improved.

Critical weaknesses
  1. The paper does not evaluate the memory under dynamic updates. Given the high cost of building the memory, its behavior under incremental updates is particularly important in practice. It would be useful to examine how the knowledge graph and temporal-semantic tree evolve as new user–agent interaction episodes arrive, including when the memory is built incrementally from scratch. Robustness to updates is critical for using the system as agent memory rather than as a static knowledge base. However, the quality of this evolution is not evaluated. Reliable behavior cannot be taken for granted, as updates to a knowledge graph can lead to degradation.

  2. The ablation study is limited to LoCoMo. It would be more informative to conduct it on all three datasets used in the main evaluation.

  3. The additive scoring function in Eq. (3) raises concerns. It combines semantic similarity, temporal relevance, and memory robustness. A query may require a specific time interval but use wording that matches many recurring situations, such as “Do you remember the restaurant we visited yesterday?” In such cases, a memory outside the requested interval could be more semantically similar to the query than the relevant memory. The additive formulation allows semantic similarity to compensate for a temporal mismatch. This is particularly concerning because semantic similarity receives a much higher weight of 0.7. Multiplying the semantic and temporal components may be more appropriate, since the retrieved evidence should satisfy both requirements. Further ablations of this score would be useful, including a comparison with semantic similarity alone and removing each component in turn.

Questions for authors
  1. Could you discuss how the knowledge graph and temporal-semantic tree evolve as new user–agent interactions arrive, including when memory is built incrementally from scratch? How does the architecture address potential degradation under repeated updates?

  2. Could you extend the ablation study to LongMemEval-S and REALTALK to assess whether the component contributions observed on LoCoMo hold across datasets?

  3. Could you justify the additive scoring function in Eq. (3), particularly for queries with explicit time constraints? Could you include ablations of the scoring function (e.g., comparing the full score with semantic similarity alone or testing a multiplicative combination of the temporal and semantic components)?

Reviewer_EPRxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed and studied a new approach to memory mechansim design for conversational agents, called H-Mem, which uses a hybrid structure with both tree and graph representations to model the evoluation of agent memory over a long period of time as well as enable efficient retrieval. The tree structure reflects evolution of memory information with a high-level node providing a longer-term summary. The graph structure represents entities and relations. Experiment results show that H-Mem achieved similar results to the state-of-the-art methods.

Critical strengths
  • How to design an effective memory mechanism is an important yet difficult problem in designing intelligent agents. There are many methods proposed, but it remains unclear which method(s) are the best. The proposed approach in this paper adds yet another approach.
  • The use of a tree structure to model evolution of information over a long period is an interesting new idea.
  • The detailed matching criteria including three dimensions, semantic similarity, temporal overlap, and memory robustness, are interesting technical contributions. The memory robustness could be more broadly interpreted as some kind of prior from Bayesian reasoning perspective.
Critical weaknesses
  • The review of related work appears to be incomplete. For example, the following ICML'26 paper proposed a hybrid structure for memory organization as well, which is shown to work well for multiple tasks: PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents, Ke Yang, Zixi Chen, Xuan He, Jize Jiang, Michel Galley, Chenglong Wang, Jianfeng Gao, Jiawei Han, ChengXiang Zhai, Proceedings of ICML'26. https://openreview.net/pdf?id=NWKaQIKoGp A discussion of the relation with this paper would be useful, especially for the purpose of highlighting the novelty of H-Mem.
  • H-Mem doesn't always outperform the existing approaches, but there seems to be no attempt to look into the reason.
  • The design of H-Mem implies that the original memory information would not be used at the time of retrieval, but the construction of H-Mem might also not retain all the important information from the original memory data. If there's any critical information missing, which may be part of an answer to a question, it would make it impossible for the agent to correctly answer such a question.
Questions for authors
  • How is work different from PlugMem, which can also be regarded as using a hybrid structure?
  • What if the construction of H-Mem failed to include some critical information that might be important for answering a question? How can you guarantee that no answer information would be missed?
Reviewer_1jRCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H-Mem, a memory system for LLM agents that combines a temporal-semantic tree and a knowledge graph. The tree organizes memories by time and topic, while the graph captures relationships between entities. The system also uses query decomposition and memory-aware retrieval to find relevant information.

The authors evaluate H-Mem on three benchmarks: LoCoMo, LongMemEval-S, and REALTALK. The results show better overall F1 and LLM-Judge Accuracy than the compared methods. Ablation studies also show that both the tree and graph help improve performance.

Critical strengths
  • Good performance across multiple benchmarks.
  • Clear design with useful ablation studies.
Critical weaknesses
  • Limited novelty. Most components of H-Mem, such as hierarchical memory, knowledge graphs, query decomposition, and memory decay, have been studied in previous work. The main contribution seems to be combining these existing methods. The paper should better explain what is new compared with systems such as EverMemOS and MemTree.

  • The experiments do not fully explain where the improvements come from. H-Mem includes several components beyond the hybrid memory structure, such as query decomposition, reranking, and additional retrieval. While the ablation study removes some components, it is still unclear how much improvement comes from the memory structure itself. A comparison using the same retrieval pipeline with different memory structures would make the results more convincing.

  • Memory evolution is not directly evaluated One main claim is that H-Mem can gradually turn short-term memories into long-term memories. However, the experiments mainly focus on QA accuracy. There is limited evidence showing how well the system handles memory updates, conflicting information, or changing user preferences over time. These are important for real-world agent memory.

  • Limited scalability analysis. H-Mem requires several steps to build and maintain its memory, which leads to higher indexing and storage costs. The paper reports these costs but does not show how they change as memory grows. It is unclear whether the system can efficiently support agents with very long interaction histories.

  • Limited evaluation of result stability. The paper reports repeated experiments, but the retrieval plans are fixed across runs. This does not capture possible changes caused by memory construction or query planning. Also, the paper does not report statistical significance when comparing with other methods. More evidence would help confirm that the improvements are reliable.

Questions for authors
  • What is the main technical difference between H-Mem and existing memory systems such as EverMemOS? Which part of the method is the key new contribution?

  • Can the authors compare different memory structures using the same retrieval and reasoning pipeline? This would help show how much improvement comes from the hybrid structure.

  • How does H-Mem handle memory updates and conflicts over time? Can the authors provide experiments that directly measure the quality of long-term memory consolidation?

  • How do indexing time, storage cost, and retrieval latency change as the number of memories increases? What is the cost of adding new memories to an existing index?

  • Can the authors report results with independently generated retrieval plans and provide statistical significance tests against the strongest baselines?

Reviewer_Kwd3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H-Mem, an agent memory mechanism with three main contributions. The first is a hybrid index: a temporal-semantic tree that consolidates atomic conversation events into provenance-preserving long-term memories across day, week, month and year windows, paired with an entity knowledge graph that captures multi-hop relations across fragments. The second is a retrieval method that exploits this structure. An LLM planner decomposes queries into scoped sub-queries, evidence is gathered through graph expansion followed by bottom-up tree traversal, and candidates are ranked by semantic similarity, temporal relevance and an Ebbinghaus-inspired recency/reinforcement score. The third is a targeted missing-information query that recovers bridge evidence when the first retrieval pass is insufficient.

Critical strengths
  1. Gains are consistent and match the design. H-Mem improves over strong baselines on all three benchmarks, including the crowdsourced REALTALK dataset. The largest improvements come where the design predicts they should: temporal reasoning and knowledge-update questions (from the tree), and multi-hop questions (from graph expansion).

  2. The system is well documented and reproducible. Consolidation preserves provenance: original fragments stay as an immutable evidence layer, and consolidated memories record their sources plus stable/evolving/conflicting states. This addresses a real weakness of summarization-based memory, which loses or overwrites facts.

Critical weaknesses
  1. The conceptual novelty is limited. Each component closely follows existing work: hierarchical tree memory (MemTree, MemOS, EverMemOS), temporal knowledge graphs (Zep), graph RAG (HippoRAG) and agentic query decomposition. The paper argues the contribution lies in coupling tree-based evolution with graph-based reasoning, but no experiment shows the coupling does more than the two indexes would independently

  2. (Minor) In evaluation section, H-Mem’s F1 is lower with the stronger GPT-4.1-mini backbone than with GPT-4o-mini, while every baseline improves. This is unexplained and may come from the LLM answer-simplification step.

Questions for authors
  1. Baseline provenance. Were all baselines re-run in your framework, or were any numbers taken from prior papers?
  2. Backbone inversion. H-Mem’s overall F1 is higher with GPT-4o-mini (56.06) than with GPT-4.1-mini (55.58), and single-hop F1 drops from 60.20 to 57.67, while every baseline improves with the stronger model. What explains this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.