acceptodds
Under review as a conference paper at ICLR 2027

HiMem: A Hierarchical Multi-Granular System for Long-term Memory

Abstract

Long-term memory is critical for interactive agents to retain and utilize information accumulated over extended conversations. However, structurally organizing fragmented memories and accurately retrieving them remain significant challenges, particularly when relevant evidence lacks direct semantic similarity to the query. Existing systems still struggle to build a structural framework for evolving interaction records and to leverage structural information to bridge the semantic gaps. To address this challenge, we propose **HiMem**, a **Hi**erarchical Multi-granular **Mem**ory System with structure-aware multi-route retrieval, which organizes evolving memories and source records across multiple granularities and exploits their structural relations to recover complementary evidence. Specifically, HiMem organizes memories around verified anchors and constructs a multi-granular graph that preserves provenance of source records, building a foundation for multi-route retrieval. During retrieval, HiMem collects complementary candidates through multi-route retrieval and re-ranks them under a unified scoring mechanism that integrates semantic and structural signals. Experiments on LoCoMo show that HiMem achieves 92.60% average accuracy, outperforming the leading memory baseline, EverMemOS, by 2.15 percentage points. Furthermore, HiMem matches the performance of the full-context prompt method, while consuming merely 20.56% of the token cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Long-Term Memory, Memory Graph, Interactive Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_L1MFOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Long-term memory is crucial for chatbots and LLM agents, especially in terms of memory organization and accurate retrieval. The authors claim that "Existing systems still struggle to build a structural framework for evolving interaction records and to leverage structural information to bridge the semantic gaps". They further propose HiMem, a Hierarchical Multi-granular Memory System with structure-aware multi-route retrieval, which organizes evolving memories and source records across multiple granularities and exploits their structural relations to recover complementary evidence. HiMem achieves 92.6% on LoCoMo and 88% on LongMemEval-S, surpassing the baselines listed in the manuscript.

Critical strengths
  1. The topic is important. AI memory remains a critical component for chatbots and LLM agents.
  2. The reported LoCoMo results, component ablations, and retrieval-budget analysis provide useful initial evidence of the system’s effectiveness within the evaluated settings.
Critical weaknesses
  1. Motivation and contribution. While I agree that memory is an important topic. The authors might overstate some limitations of existing approaches and does not sufficiently establish its contribution relative to closely related work. In Introduction, the authors point out two challenges. (1) Existing memory construction relies heavily on semantic similarity. HiMem has similarity filter and LLM-verifier. This verification can reject semantically similar but distinct occurrences, but cannot recover same-occurrence matches excluded during candidate generation. Moreover, LLM-based event identity checking and consolidation have already been explored in prior memory frameworks. (2) Existing memory retrieval lacks effective mechanisms to retrieve complementary evidences from heterogeneous sources. I think hierarchical and graph-based memory methods have already explored complementary semantic and structural retrieval, relational expansion, and evidence collection across multiple granularities. Given that the motivation is not enough explained, the improvement seems marginal. While the authors claimed to be SOTA on LoCoMo using GPT-4.1-mini. EverMemOS, using 2.3k tokens, achieves 93.05, surpassing Himem (92.6 at k = 20).

  2. Reproduction. As introduced in Sec 3.1 and Table 2, there are many components in HiMem. Critical components include event nodes, atomic memories, relations, etc. However, the manuscript does not explore enough details about these components. Sec 3.1 and Appendix C merely define these components formally. However, it is hard to know how these components look like, and how they are obtained. The authors should share some real cases and put the construction prompt, at least in the appendix.

  3. Organization (minor). This manuscript should improve the organization to put important information in the main text (e.g. LME-S results, method details).

Questions for authors
  1. Users may frequently alternate between unrelated topics within a conversation, leaving information about the same event distributed across non-contiguous segments. How robust is HiMem’s memory construction under such topic interleaving, particularly when later utterances contain implicit references to earlier segments?
2 comments
AuthorsOpenReview ↗

Response to Reviewer L1MF

We sincerely appreciate the reviewer's valuable comments on the contribution, implementation details, and handling of fragmented conversations.

Weaknesses & Questions:

W1 (Motivation, novelty, and contribution)
The paper may overstate the limitations of existing memory systems and does not sufficiently establish its contributions relative to prior work. Specifically, (1) HiMem's similarity-based candidate generation may miss memories belonging to the same occurrence before LLM verification, and occurrence verification and consolidation have already been explored in prior work; (2) hierarchical memory graphs and complementary multi-route retrieval are not novel. The reviewer also questions the marginal improvement, citing EverMemOS's originally reported 93.05% accuracy.

We sincerely appreciate this observation. Our main contribution is to connect occurrence-verified event organization with retrieval of both atomic memories and source records. We use the same anchors across construction, consolidation, and recall. Our ablations show the value of these components (Table 2), and our evidence analysis shows the additional support provided by event-mediated retrieval at the evaluated cutoff (Figure 4/Table 7).

We agree with your point about candidate recall. We check up to five event candidates above similarity 0.80 (Appendix D.1). If the correct anchor is missed, we may create a separate event for the same occurrence. Our verification step helps avoid incorrect merges among retrieved candidates; it cannot recover an anchor that candidate search has missed.

Our EverMemOS comparison is 92.60% versus 90.45%, while its original paper reports 93.05%. As we could not identify a release explicitly tied to the paper, we selected the available version closest in time to it, v1.1.0 (commit 77c416c), and evaluated it under our common answer and judge pipeline (Appendices B.1–B.2).

W2 (Reproducibility and insufficient implementation details)
The manuscript introduces many components, including event nodes, atomic memories, and typed relations, but lacks sufficient implementation details and concrete examples. The reviewer requests real cases illustrating how these components are constructed and the actual memory-construction prompts, at least in the appendix.

We sincerely appreciate this helpful suggestion. We describe the memory representations, event assignment, relation types, consolidation rules, and scoring equations in Section 3.1 and Appendices C–E. Table 10 provides retrieval examples, but we agree that actual prompts and a construction example would make our method easier to reproduce. We can supplement these descriptions with the construction prompts and an example showing how a source segment becomes atomic memories, event assignments, and relations.

W3 (Paper organization and presentation)
The manuscript should improve its organization by moving important information, such as LongMemEval-S results and essential method details, from the appendix to the main text.

Thank you for this suggestion. We will add a short summary of our LongMemEval-S result (88.00%) to the main text and clearly identify the sources of the baseline scores. We will also explain the key assignment steps more directly: retrieve candidate events, check occurrence compatibility, and assign the memory or create a new anchor. We will keep the full prompts and parameter lists in the appendix.

Q1 (Robustness to interleaved topics and implicit references)
Users may frequently alternate between unrelated topics within a conversation, leaving information about the same event distributed across non-contiguous segments. How robust is HiMem's memory construction under such topic interleaving, particularly when later utterances contain implicit references to earlier segments?

Thank you for this question. We search the accumulated event store for each new memory, so memories from separate segments can join the same earlier event. We also use neighboring events during consolidation to help resolve references and preserve source records to retain the original context.

We can still miss a connection if extraction fails to resolve a reference or candidate search misses the correct event. Due to time and computational constraints, we have not separately evaluated topic interleaving or cross-segment references.

Thank you for your response. While I appreciate the clarifications the authors make, they do not help improve the quality of this manuscript. As I have made clear, the authors should further discuss existing work to clarify their contribution and avoid overstatement. I regret that neither the response nor the manuscript addresses this concern. Besides, the proposed method leaves key problems unsolved. Given the authors’ response and the issues raised by other reviewers, I will change my score to lean towards rejection.

Reviewer_D4LCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies long-term memory in LLM agents. The authors argue that retrieving memories is difficult when the evidence lacks direct semantic similarity to the query. The paper argues that current systems struggle with organizing evolving memories and exploiting their structural relations. The paper proposes HiMem, a hierarchical multi-granular memory system, which organizes evolving memories and source records across multiple granularities and exploits their relations. It does so by constructing a graph that preserves provenance to the original records. To address the semantic similarity shortcoming, the method collects complementary candidates and re-ranks them using a scoring mechanism that integrates different signals. Experimentally, the authors claim the method performs favorably compared with prior methods and matches the performance of full-context methods while requiring fewer answer-stage input tokens.

The main contributions seem to be (1) explicit verification of memories' correspondence to the same occurrence, and (2) a particular way of combining and reranking results from multiple retrievers.

Critical strengths
  1. The problem is interesting - addressing long-term memory is relevant.
  2. The experimental evaluation is comprehensive.
Critical weaknesses

The biggest concern with this work is relatively limited performance boost compared to the technical complexity it introduces.

  1. The novelty over prior work is limited. EverMemOS already uses hierarchical memory and multi-route retrieval, and Zep combines semantic, lexical, and graph retrieval.
  2. The improvements over EverMemOS are small (92.60 vs 90.45). Also the original EverMemOS paper reports 93.05, which is higher than what this paper reports (with fewer tokens too). It's unclear why this discrepancy exists.
  3. In Figure 3 only HiMem's retrieval budget is varied. But EverMemOS also compared the accuracy-vs-token tradeoff; the comparison should be explicit.
  4. Dense + BM25 already achieves 91.30, compared with 92.60 for the full method. I feel that the relatively little performance boost may not justify its technical complexity.
  5. The hyperparameter selection procedure is unclear, particularly since no validation split is used and the retrieval budget appears to be selected using test performance. This information should be included in the paper.
Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer D4LC (1/2)

We sincerely appreciate the reviewer's constructive feedback and recognition of the experimental evaluation.

Weaknesses & Questions:

W1 (Limited novelty and contribution)
The novelty over prior work is limited. EverMemOS already uses hierarchical memory and multi-route retrieval, while Zep combines semantic, lexical, and graph retrieval. The reviewer questions whether HiMem introduces sufficient technical novelty.

We sincerely appreciate this comment. Our innovation is to use occurrence-verified event anchors to connect memory construction, consolidation, and recall, together with separate ranking of atomic memories and original source records. This design lets us use the memory structure directly to recover complementary evidence at different levels.

In our ablations, removing event structure reduces accuracy from 92.60% to 90.84%, and removing verification gives 91.62% (Table 2). Our event-mediated route also supports 18.45% of annotated evidence instances without a matching dense-sourced memory at the evaluated cutoff (Figure 4/Table 7). We also acknowledge that our main-text description of Zep is too broad and will correct it to reflect its hybrid retrieval, as described in Appendix B.1.

W2 (Limited performance improvement over EverMemOS)
HiMem achieves only a modest improvement over EverMemOS (92.60 vs. 90.45). Moreover, the original EverMemOS paper reports 93.05% accuracy with fewer tokens, which is higher than HiMem's reported result. The reviewer requests an explanation for this discrepancy.

We sincerely appreciate this comment. HiMem improves accuracy from 90.45% to 92.60% in our LoCoMo evaluation. This 2.15-point gain corresponds to about 22.5% relative error reduction. With the baseline already above 90%, we consider this a meaningful improvement over a strong memory system.

LoCoMo also has reported annotation errors, including incorrect dates, speaker assignments, and unsupported reference answers. These issues make scores at this accuracy level harder to interpret and highlight the importance of comparing systems under the same evaluation settings. LoCoMo annotation reports, public audit

The original EverMemOS paper reports 93.05%. Its repository is continuously updated, and we could not identify a release explicitly tied to the paper. We therefore selected the available version closest in time to the paper, v1.1.0 (commit 77c416c), and fixed this version for evaluation. We used group-scoped agentic retrieval and top-ten episodic memories under our common answer and judge pipeline (Appendices B.1–B.2). Our reported gain is measured against this reproduction.

W3 (Accuracy–token trade-off comparison)
Figure 3 varies only HiMem's retrieval budget, while EverMemOS is evaluated under its default setting. Since the original EverMemOS paper also reports an accuracy–token trade-off curve, the reviewer suggests explicitly comparing the two curves.

Thank you for this suggestion. We agree that comparing the two accuracy–token curves under a unified protocol would be more informative. We first note that HiMem's advantage is not solely due to a larger answer context: at (K,C)=(10,3), HiMem achieves 91.10% with 2,796 answer-input tokens, outperforming EverMemOS's evaluated default of 90.45% with 3,198 tokens (Figure 3/Table 9).

The curve in the original EverMemOS paper was obtained under its own retrieval and evaluation settings, which differ from our common answer and judge pipeline, so directly overlaying the two curves would not be a controlled comparison. We have not yet completed a full retrieval-budget sweep for EverMemOS under our protocol, and we will add this comparison in the revision.

W4 (Performance gain versus system complexity)
Dense + BM25 already achieves 91.30% accuracy, compared with 92.60% for the full HiMem system. The reviewer questions whether the additional 1.30-point improvement justifies the complexity introduced by event verification, graph construction, consolidation, and structure-aware retrieval.

We appreciate this practical question. HiMem improves accuracy from 91.30% for dense + BM25 to 92.60%, or about 14.9% relative error reduction. Answer-input tokens increase by only about 2.3%, from 4,445 to 4,548. This is an additional accuracy gain over a strong hybrid-retrieval baseline at a similar answer-context size.

Our evidence analysis shows the additional information recovered by event-mediated retrieval (Figure 4/Table 7). Our ablations also show that the retrieval routes work best in combination: dense + graph scores 86.82%, compared with 87.14% for dense alone. We agree that the added construction and operating cost also matters. We have not completed that cost comparison, so we can currently quantify the accuracy gain and answer-input tokens.

AuthorsOpenReview ↗

Response to Reviewer D4LC (2/2)

W5 (Hyperparameter selection and potential test-set tuning)
The hyperparameter selection procedure is unclear. In particular, no validation split is specified, and the retrieval budget appears to be selected based on test-set performance. The reviewer requests clarification on how hyperparameters and retrieval budgets were determined.

Thank you for asking us to clarify this point. The event threshold, ranking weights, and default (K,C)=(20,5) were fixed before evaluation, without using LoCoMo test accuracy. We did not use a separate validation split to choose these settings. The budget sweep was used to examine performance at different budgets, not to select the default.

We recognize that our wording in Appendix F.4 suggests that we selected the default after seeing the curve. We will correct it to state that the default was fixed before evaluation and that the budget sweep was used only to examine the accuracy–token trade-off.

Thank you for asking us to clarify this point. The event threshold, ranking weights, and default (K,C)=(20,5) were fixed before evaluation, without using LoCoMo test accuracy. We did not use a separate validation split to choose these settings. The budget sweep was used to examine performance at different budgets, not to select the default.

Could you please provide some hyperparameter sensitivity results of the impact of these K,C variables? It'd give a sense of robustness.

Reviewer_tKvWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HiMem organizes long-term conversational memory around occurrence-verified event nodes, with atomic memories, source records, provenance-preserving consolidation, and typed relations, and retrieves evidence through dense, event-mediated, relational, and lexical routes reranked under a unified semantic plus structural score. It is evaluated on LoCoMo and LongMemEval-S.

Critical strengths
  1. Occurrence verification is the one part that feels genuinely useful. Merging anchors by similarity alone conflates different events that happen to look alike, and checking participants, action, and time gets at that problem directly.
  2. The event-only evidence result is interesting. It shows the graph is doing something dense retrieval alone does not do.
  3. The system beats EverMemOS on both benchmarks and both backbones.
  4. The budget curve is decent. Gains flatten out sensibly instead of just growing with more context.
  5. HiMem marks old memories as superseded when a newer version replaces them, instead of just scoring old memories lower based on recency scores.
Critical weaknesses
  1. This is incremental work. Event graphs, hierarchical graphs, and multi-route retrieval are all already out there, and occurrence verification is a twist on that line, not a new kind of system.
  2. The SOTA claim is unproven. The main table is missing Mnemis, CompassMem, AssoMem, A-Mem, MemGAS, and HippoRAG 2, Mem0 is no longer competitive, Mnemis reports higher numbers on both benchmarks, and the main results are on GPT-4.1-mini, an old backbone.
  3. LongMemEval is reported only on final accuracy, only the small setting, and baselines taken second hand from EverMemOS, with no retrieval numbers and no medium or large setting.
  4. The key components are not validated directly. Event assignment and relation verification are only shown through ablations, and there is no importance signal, which shows up in the weaker single-session preference result.
  5. The real cost is unknown. It uses more answer tokens than EverMemOS, and there is no memory construction cost or latency at all.
  6. The judge is the same model as the main backbone, with no human check and no system-level variance.
Questions for authors
  1. Can you compare with the strongest baselines?
  2. How accurate is event assignment and relation verification on their own, not just in the downstream ablation?
  3. Can you rerun the LongMemEval-S baselines under your own code and judge, with medium and large settings?
  4. How sensitive are the results to the ranking weights and the event similarity threshold?
  5. What are the construction token cost and per-query latency?
2 comments
AuthorsOpenReview ↗

Response to Reviewer tKvW (1/2)

We sincerely thank the reviewer for the constructive feedback. We appreciate the reviewer’s recognition of the limitations and open questions in our evaluation, and the suggestions for strengthening the empirical validation of HiMem. We address each concern below and clarify the scope of our current contributions and experimental results.

Weaknesses & Questions:

W1
The work is considered incremental, with limited novelty and contribution. Event graphs, hierarchical memory organization, and multi-route retrieval have been explored in prior work. Occurrence verification is viewed as an improvement to existing approaches rather than a fundamentally new system.

We sincerely appreciate this comment. HiMem's distinctive contribution is the connection between occurrence-verified memory organization and structure-aware evidence retrieval. It checks whether memories describe the same occurrence, then uses the verified event anchors across construction, consolidation, and recall. It also retrieves and ranks both atomic memories and original source records, combining precise facts with their context. Together, these shared anchors and complementary evidence units form HiMem's distinctive design relative to prior work.

Table 2 supports this design: removing event structure reduces accuracy from 92.60% to 90.84%, and removing occurrence verification gives 91.62%. At the evaluated cutoff, event-mediated retrieval also supports 18.45% of annotated evidence instances without a matching dense-sourced memory (Figure 4/Table 7). These results show how the event structure helps recover additional evidence.

W2 and Q1
The SOTA claim is not sufficiently supported. The main comparison omits Mnemis, CompassMem, AssoMem, A-Mem, MemGAS, and HippoRAG 2. The reviewer argues that Mem0 is no longer competitive, that Mnemis reports stronger results, and that the GPT-4.1-mini setting is not sufficiently up to date.
Q1: Can you compare HiMem against the strongest recent baselines?

Thank you for suggesting additional comparisons. Alongside Mem0, our evaluation includes EverMemOS, a strong recent baseline that reports 93.05% on LoCoMo with GPT-4.1-mini. Our comparison therefore already includes a competitive recent memory system. EverMemOS paper

Futhermore, Mnemis has released Global Selection code and evaluation results, but we did not find a complete pipeline for rebuilding both memory graphs from raw conversations and reproducing its evaluation in the materials inspected. Comparing it under our settings would require additional implementation work. We have not run Mnemis, CompassMem, or the other suggested methods in our pipeline. Mnemis repository

As for the backbone model selection, GPT-4.1-mini is widely used by recent work, including Mnemis and EverMemOS. We additionally test Gemini-3.8-flash: HiMem achieves 93.18%, compared with 92.14% for our EverMemOS reproduction (Table 1). The gains with both backbones show that HiMem's benefit extends across the two tested models. Mnemis paper

W3 and Q3 (LongMemEval evaluation)
The LongMemEval evaluation reports only final accuracy on LongMemEval-S, with comparison numbers taken from EverMemOS (and originally MemOS). It lacks independently rerun baselines, retrieval-level measurements, and evaluations on the medium/large settings.
Q3: Can you rerun the LongMemEval-S baselines using your own pipeline and judge, and evaluate medium/large settings?

Thank you for this suggestion. As stated in Appendix F.1, the LongMemEval-S baseline scores are taken from prior work. Due to time and computational constraints, we did not rerun these baselines or add the requested retrieval-level and larger-history evaluations.

AuthorsOpenReview ↗

Response to Reviewer tKvW (2/2)

W4 and Q2 (Direct validation of memory construction)
The key construction components are validated mainly through downstream ablations rather than direct intrinsic metrics. The reviewer requests independent measurements of event assignment and relation verification accuracy. The review also questions the absence of an explicit memory-importance signal, relating it to the weaker single-session preference result.
Q2: How accurate are event assignment and relation verification on their own, independently of downstream QA performance?

We sincerely appreciate this suggestion. We evaluated these components through QA ablations: removing occurrence verification gives 91.62%, and removing typed relations gives 91.75%, compared with 92.60% for our full method (Table 2). We have not separately measured event-assignment or relation-label accuracy. These evaluations require additional annotations, which we have not collected within the available time and resources.

Thank you for suggesting an explicit importance signal. This may help with preference retrieval. HiMem currently achieves 93.33% on preference questions, matching EverMemOS and below MemOS's 96.67% (Table 6). Further experiments would be needed to determine whether adding this signal improves our results.

Q4 (Hyperparameter sensitivity)
The reviewer questions whether the results are sensitive to ranking weights and the event-similarity threshold.
Q4: How sensitive are the results to the ranking weights and the event similarity threshold?

Thank you for this suggestion. We fixed the event threshold, ranking weights, and default retrieval budgets before evaluation, without using LoCoMo test accuracy. We list these settings in Appendix E/Table 5 and evaluate different retrieval budgets in Figure 3/Table 9. Due to time and computational constraints, we have not conducted a separate sensitivity analysis of the event threshold or ranking weights.

W5 and Q5 (End-to-end cost and latency)
The reviewer argues that the real operating cost is unclear: HiMem uses more answer-stage input tokens than EverMemOS under their default configurations, while memory-construction cost and end-to-end latency are not reported.
Q5: What are the construction-token cost and per-query latency?

We appreciate this question. HiMem's default setting uses more answer-input tokens than EverMemOS: 4,548 versus 3,198. At (K,C)=(10,3), HiMem achieves 91.10% with 2,796 tokens, compared with EverMemOS's evaluated default of 90.45% with 3,198 tokens (Figure 3/Table 9). Thus, HiMem also provides an accuracy gain with a smaller answer context at this setting.

Our cost analysis covers answer-stage input tokens. We have not completed a comparison of construction cost or per-query latency. Our API-based timing also varies with service response times and network conditions, so we cannot provide a reliable latency comparison from the current measurements.

W6 (Judge reliability and system-level variance)
The reviewer questions the use of GPT-4.1-mini as both the primary backbone and correctness judge, and notes the absence of human validation and system-level run-to-run variance.

Thank you for raising this concern. All methods in our LoCoMo comparison use the same judge setup, with method identities hidden and majority voting over three calls (Appendix B.2). HiMem also outperforms our EverMemOS reproduction when Gemini-3.8-flash is the backbone and GPT-4.1-mini is the judge. The gain therefore also appears when the backbone and judge are different models.

We also repeated judging on a fixed set of archived answers and report the results in Appendix F.7/Table 11. This checks judge consistency on a different answer snapshot from Table 1. We have not conducted human validation or repeated full-system runs, so our current evaluation does not measure these sources of uncertainty.

Comments on the paper

1 comment
Area_Chair_ttqFOpenReview ↗

Discussion Window Closing in 2 days. Please Respond to the Author's Rebuttal!

Dear Reviewers,

The authors have thoughtfully responded to your reviews and the discussion window is coming to a close on November 18th. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.