- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Long-term memory is crucial for chatbots and LLM agents, especially in terms of memory organization and accurate retrieval. The authors claim that "Existing systems still struggle to build a structural framework for evolving interaction records and to leverage structural information to bridge the semantic gaps". They further propose HiMem, a Hierarchical Multi-granular Memory System with structure-aware multi-route retrieval, which organizes evolving memories and source records across multiple granularities and exploits their structural relations to recover complementary evidence. HiMem achieves 92.6% on LoCoMo and 88% on LongMemEval-S, surpassing the baselines listed in the manuscript.
Critical strengths
- The topic is important. AI memory remains a critical component for chatbots and LLM agents.
- The reported LoCoMo results, component ablations, and retrieval-budget analysis provide useful initial evidence of the system’s effectiveness within the evaluated settings.
Critical weaknesses
-
Motivation and contribution. While I agree that memory is an important topic. The authors might overstate some limitations of existing approaches and does not sufficiently establish its contribution relative to closely related work. In Introduction, the authors point out two challenges. (1) Existing memory construction relies heavily on semantic similarity. HiMem has similarity filter and LLM-verifier. This verification can reject semantically similar but distinct occurrences, but cannot recover same-occurrence matches excluded during candidate generation. Moreover, LLM-based event identity checking and consolidation have already been explored in prior memory frameworks. (2) Existing memory retrieval lacks effective mechanisms to retrieve complementary evidences from heterogeneous sources. I think hierarchical and graph-based memory methods have already explored complementary semantic and structural retrieval, relational expansion, and evidence collection across multiple granularities. Given that the motivation is not enough explained, the improvement seems marginal. While the authors claimed to be SOTA on LoCoMo using GPT-4.1-mini. EverMemOS, using 2.3k tokens, achieves 93.05, surpassing Himem (92.6 at k = 20).
-
Reproduction. As introduced in Sec 3.1 and Table 2, there are many components in HiMem. Critical components include event nodes, atomic memories, relations, etc. However, the manuscript does not explore enough details about these components. Sec 3.1 and Appendix C merely define these components formally. However, it is hard to know how these components look like, and how they are obtained. The authors should share some real cases and put the construction prompt, at least in the appendix.
-
Organization (minor). This manuscript should improve the organization to put important information in the main text (e.g. LME-S results, method details).
Questions for authors
- Users may frequently alternate between unrelated topics within a conversation, leaving information about the same event distributed across non-contiguous segments. How robust is HiMem’s memory construction under such topic interleaving, particularly when later utterances contain implicit references to earlier segments?
2 comments
Thank you for your response. While I appreciate the clarifications the authors make, they do not help improve the quality of this manuscript. As I have made clear, the authors should further discuss existing work to clarify their contribution and avoid overstatement. I regret that neither the response nor the manuscript addresses this concern. Besides, the proposed method leaves key problems unsolved. Given the authors’ response and the issues raised by other reviewers, I will change my score to lean towards rejection.
Response to Reviewer L1MF
We sincerely appreciate the reviewer's valuable comments on the contribution, implementation details, and handling of fragmented conversations.
Weaknesses & Questions:
We sincerely appreciate this observation. Our main contribution is to connect occurrence-verified event organization with retrieval of both atomic memories and source records. We use the same anchors across construction, consolidation, and recall. Our ablations show the value of these components (Table 2), and our evidence analysis shows the additional support provided by event-mediated retrieval at the evaluated cutoff (Figure 4/Table 7).
We agree with your point about candidate recall. We check up to five event candidates above similarity 0.80 (Appendix D.1). If the correct anchor is missed, we may create a separate event for the same occurrence. Our verification step helps avoid incorrect merges among retrieved candidates; it cannot recover an anchor that candidate search has missed.
Our EverMemOS comparison is 92.60% versus 90.45%, while its original paper reports 93.05%. As we could not identify a release explicitly tied to the paper, we selected the available version closest in time to it, v1.1.0 (commit 77c416c), and evaluated it under our common answer and judge pipeline (Appendices B.1–B.2).
We sincerely appreciate this helpful suggestion. We describe the memory representations, event assignment, relation types, consolidation rules, and scoring equations in Section 3.1 and Appendices C–E. Table 10 provides retrieval examples, but we agree that actual prompts and a construction example would make our method easier to reproduce. We can supplement these descriptions with the construction prompts and an example showing how a source segment becomes atomic memories, event assignments, and relations.
Thank you for this suggestion. We will add a short summary of our LongMemEval-S result (88.00%) to the main text and clearly identify the sources of the baseline scores. We will also explain the key assignment steps more directly: retrieve candidate events, check occurrence compatibility, and assign the memory or create a new anchor. We will keep the full prompts and parameter lists in the appendix.
Thank you for this question. We search the accumulated event store for each new memory, so memories from separate segments can join the same earlier event. We also use neighboring events during consolidation to help resolve references and preserve source records to retain the original context.
We can still miss a connection if extraction fails to resolve a reference or candidate search misses the correct event. Due to time and computational constraints, we have not separately evaluated topic interleaving or cross-segment references.