- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the organization and retrieval of long-term conversational memory and proposes SUE-Mem. Its core idea is to allow an event to belong to multiple storylines while maintaining and updating each storyline independently. Adaptive splitting and hierarchical retrieval further support evidence collection for question answering. Using GPT-4o-mini as the backbone, the authors evaluate the framework on LoCoMo and LongMemEval-S through comparisons with existing methods, component ablations, and efficiency analyses.
Critical strengths
-
A well-motivated algorithmic design for conversational memory. SUE-Mem allows an event to belong to multiple independently maintained storylines, with adaptive splitting and hierarchical retrieval. This directly addresses the need to preserve different narrative contexts around shared events. The component ablations support the contribution of key design choices to question-answering performance.
-
Demonstrated effectiveness within a complete memory pipeline. Experiments on LoCoMo and LongMemEval-S show the usefulness of the framework with GPT-4o-mini. The paper describes an end-to-end pipeline for memory construction, maintenance, and retrieval, and reports query-time efficiency measurements. Together, these provide evidence of practical value beyond the conceptual design, within the evaluated model setting.
Critical weaknesses
-
Limited methodological novelty beyond framework integration. The overlapping-storyline formulation is a useful design, but the overall contribution appears primarily to lie in organizing and integrating LLM-based memory operations, including event assignment, summary maintenance, splitting, and retrieval. The methodological advance over existing structured-memory approaches therefore seems incremental. While the ablations demonstrate the usefulness of individual components, they do not by themselves establish the distinctiveness of the proposed mechanism. This limits the strength of the paper’s methodological contribution.
-
The framework’s benefits remain unverified beyond a single backbone. All LLM-based components use GPT-4o-mini, leaving it unclear whether the observed gains persist with a newer model offering comparable inference cost or latency. This is not a request to evaluate the strongest available model, but to establish whether the framework retains meaningful value as backbone capabilities improve. Evaluating a representative alternative backbone with simple baselines, the full framework, and key ablations would strengthen the evidence. Without this, my confidence in the broader applicability of the framework remains limited.
Questions for authors
Could the authors provide an additional evaluation using a newer model with inference cost or latency comparable to GPT-4o-mini, comparing the full framework, a simple retrieval baseline, and key ablations? This would help establish whether the framework’s gains persist as model capabilities improve. Please clarify which components use the replacement model, particularly whether it is used only for answer generation or also for memory construction and maintenance.
4 comments
Thank you for your time and effort in reviewing our paper. Your feedback is very valuable for improving this work.
Since the author-reviewer discussion period will end in about one day, we would appreciate it if you could take a look at our rebuttal when you have a moment and let us know whether your concerns have been addressed.
Thank you again for your time and suggestions.
Best regards, The authors
Thank you for the additional experiments and clarifications. The new-backbone results help address my main concern. I also recognize the other reviewers’ concerns about methodological novelty, evaluation coverage, and memory-construction costs, and appreciate the added matched-budget comparison and human evaluation. Considering these points, I will maintain my rating at this stage.
Thank you very much for your valuable review. We greatly appreciate that you recognize the value of our newly added experiments.
Regarding the memory‑construction cost: we admit that our method brings a slightly higher construction overhead. Nevertheless, our approach achieves well‑balanced overall performance across different settings, which demonstrates its practical merit.
In terms of methodological novelty, our key contribution is that we are, to our knowledge, the first to propose the mechanism where sub‑events can influence multiple storylines, which forms the core novelty of this work.
We have elaborated on these points in the revised discussion section. Thank you again for your insightful comments.
Thank you for your careful reading and constructive suggestions. We appreciate your recognition of the motivation for overlapping storylines and the value of evaluating the full memory pipeline. Your question about the backbone prompted us to run an additional comparison on LoCoMo.
Q1. Do the gains persist with a different backbone, and which modules use it?
We repeated the LoCoMo comparison with DeepSeek-V4.1-Flash for SUE-Mem, StructMem, and MRAgent. The change was not limited to answer generation. All three methods used the new backbone for memory construction and query-time retrieval. For SUE-Mem, this includes the LLM judgments used to assign events to storylines and maintain those storylines over time.
With the new backbone, SUE-Mem still leads on all three Multi-Hop metrics. Its Multi-Hop J reaches 87.23 and its Overall J reaches 88.08, exceeding the strongest baseline by 7.41 and 1.86 points, respectively. Section 4.8 of the revised manuscript reports the experimental setup and results.
W1. What is new beyond combining existing memory operations?
The distinction lies in how SUE-Mem handles a shared event when it writes and updates memory. It stores the event once, uses the existing events in each candidate storyline to decide whether the new event belongs there, and lets the storylines that accept it evolve independently. For example, an award can close a competition storyline and satisfy a scholarship prerequisite. A later scholarship decision can then extend the scholarship storyline without changing the competition storyline.
The ablation addresses a specific question: does allowing storylines to overlap improve this system? Restricting each event to a single storyline reduces Multi-Hop J from 82.62 to 69.29 and Overall J from 81.56 to 73.98. This is the largest drop among the variants we tested. Sections 3.2 to 3.4 explain the membership and update rules, and Section 4.5 and Appendix D.1 report the ablation results.
W2. Is the benefit specific to GPT-4o-mini?
The Multi-Hop gains persist across both backbones tested. With DeepSeek-V4.1-Flash, SUE-Mem improves on the strongest baseline for each Multi-Hop metric by 5.54 points on F1, 2.69 on BLEU-1, and 7.41 on J. The consistent advantage across all three metrics shows that the Multi-Hop result is not specific to GPT-4o-mini. The full results are in Section 4.8 of the revised manuscript.