acceptodds
Under review as a conference paper at ICLR 2027

SUE-Mem: Logic-Guided Overlapping Event Storylines for Long-Term Conversational Memory

Abstract

Long-term dialogue agents must retrieve facts across sessions while distinguishing event relationships from topical similarity. We introduce SUE-Mem, a memory framework that organizes temporally grounded atomic events into overlapping storylines. For each new event, SUE-Mem retrieves candidate storyline summaries and uses an LLM judge with historical event evidence to assess membership. An event may join multiple independently evolving storylines, with milestone-triggered checks identifying those that should split. At query time, a hybrid retriever protects highly ranked semantic candidates and selects additional storylines using semantic, lexical, and exact-anchor scores. On LoCoMo, SUE-Mem achieves absolute gains of 14.53% in Multi-Hop and 5.59% in Overall Judge score over the strongest reported baselines. On LongMemEval, it achieves Overall F1 and Judge scores of 46.35 and 71.32. Ablations support the contributions of overlapping assignment, event-level evidence, and adaptive splitting to long-term conversational question answering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-term memory, conversational agents, memory retrieval, retrieval augmented generation, agent memory, long-context reasoning, event-based memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nnFnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the organization and retrieval of long-term conversational memory and proposes SUE-Mem. Its core idea is to allow an event to belong to multiple storylines while maintaining and updating each storyline independently. Adaptive splitting and hierarchical retrieval further support evidence collection for question answering. Using GPT-4o-mini as the backbone, the authors evaluate the framework on LoCoMo and LongMemEval-S through comparisons with existing methods, component ablations, and efficiency analyses.

Critical strengths
  1. A well-motivated algorithmic design for conversational memory. SUE-Mem allows an event to belong to multiple independently maintained storylines, with adaptive splitting and hierarchical retrieval. This directly addresses the need to preserve different narrative contexts around shared events. The component ablations support the contribution of key design choices to question-answering performance.

  2. Demonstrated effectiveness within a complete memory pipeline. Experiments on LoCoMo and LongMemEval-S show the usefulness of the framework with GPT-4o-mini. The paper describes an end-to-end pipeline for memory construction, maintenance, and retrieval, and reports query-time efficiency measurements. Together, these provide evidence of practical value beyond the conceptual design, within the evaluated model setting.

Critical weaknesses
  1. Limited methodological novelty beyond framework integration. The overlapping-storyline formulation is a useful design, but the overall contribution appears primarily to lie in organizing and integrating LLM-based memory operations, including event assignment, summary maintenance, splitting, and retrieval. The methodological advance over existing structured-memory approaches therefore seems incremental. While the ablations demonstrate the usefulness of individual components, they do not by themselves establish the distinctiveness of the proposed mechanism. This limits the strength of the paper’s methodological contribution.

  2. The framework’s benefits remain unverified beyond a single backbone. All LLM-based components use GPT-4o-mini, leaving it unclear whether the observed gains persist with a newer model offering comparable inference cost or latency. This is not a request to evaluate the strongest available model, but to establish whether the framework retains meaningful value as backbone capabilities improve. Evaluating a representative alternative backbone with simple baselines, the full framework, and key ablations would strengthen the evidence. Without this, my confidence in the broader applicability of the framework remains limited.

Questions for authors

Could the authors provide an additional evaluation using a newer model with inference cost or latency comparable to GPT-4o-mini, comparing the full framework, a simple retrieval baseline, and key ablations? This would help establish whether the framework’s gains persist as model capabilities improve. Please clarify which components use the replacement model, particularly whether it is used only for answer generation or also for memory construction and maintenance.

4 comments

Thank you for your careful reading and constructive suggestions. We appreciate your recognition of the motivation for overlapping storylines and the value of evaluating the full memory pipeline. Your question about the backbone prompted us to run an additional comparison on LoCoMo.

Q1. Do the gains persist with a different backbone, and which modules use it?

We repeated the LoCoMo comparison with DeepSeek-V4.1-Flash for SUE-Mem, StructMem, and MRAgent. The change was not limited to answer generation. All three methods used the new backbone for memory construction and query-time retrieval. For SUE-Mem, this includes the LLM judgments used to assign events to storylines and maintain those storylines over time.

With the new backbone, SUE-Mem still leads on all three Multi-Hop metrics. Its Multi-Hop J reaches 87.23 and its Overall J reaches 88.08, exceeding the strongest baseline by 7.41 and 1.86 points, respectively. Section 4.8 of the revised manuscript reports the experimental setup and results.

W1. What is new beyond combining existing memory operations?

The distinction lies in how SUE-Mem handles a shared event when it writes and updates memory. It stores the event once, uses the existing events in each candidate storyline to decide whether the new event belongs there, and lets the storylines that accept it evolve independently. For example, an award can close a competition storyline and satisfy a scholarship prerequisite. A later scholarship decision can then extend the scholarship storyline without changing the competition storyline.

The ablation addresses a specific question: does allowing storylines to overlap improve this system? Restricting each event to a single storyline reduces Multi-Hop J from 82.62 to 69.29 and Overall J from 81.56 to 73.98. This is the largest drop among the variants we tested. Sections 3.2 to 3.4 explain the membership and update rules, and Section 4.5 and Appendix D.1 report the ablation results.

W2. Is the benefit specific to GPT-4o-mini?

The Multi-Hop gains persist across both backbones tested. With DeepSeek-V4.1-Flash, SUE-Mem improves on the strongest baseline for each Multi-Hop metric by 5.54 points on F1, 2.69 on BLEU-1, and 7.41 on J. The consistent advantage across all three metrics shows that the Multi-Hop result is not specific to GPT-4o-mini. The full results are in Section 4.8 of the revised manuscript.

Thank you for your time and effort in reviewing our paper. Your feedback is very valuable for improving this work.

Since the author-reviewer discussion period will end in about one day, we would appreciate it if you could take a look at our rebuttal when you have a moment and let us know whether your concerns have been addressed.

Thank you again for your time and suggestions.

Best regards, The authors

Thank you for the additional experiments and clarifications. The new-backbone results help address my main concern. I also recognize the other reviewers’ concerns about methodological novelty, evaluation coverage, and memory-construction costs, and appreciate the added matched-budget comparison and human evaluation. Considering these points, I will maintain my rating at this stage.

Thank you very much for your valuable review. We greatly appreciate that you recognize the value of our newly added experiments.

Regarding the memory‑construction cost: we admit that our method brings a slightly higher construction overhead. Nevertheless, our approach achieves well‑balanced overall performance across different settings, which demonstrates its practical merit.

In terms of methodological novelty, our key contribution is that we are, to our knowledge, the first to propose the mechanism where sub‑events can influence multiple storylines, which forms the core novelty of this work.

We have elaborated on these points in the revised discussion section. Thank you again for your insightful comments.

Reviewer_jcvfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Problem is interesting and need of the hour. they propose a memory framework that connects related events through explicit logical relationships, allows events to participate in multiple storylines, and keeps those storylines distinct as they evolve.

Critical strengths

The idea that one event can belong to multiple independently evolving storylines is novel and should be studied.

Critical weaknesses
  1. The methodology is primarily prompt engineering where in combining these components into a pipeline that builds overlapping event storylines and splits them when needed is mostly done.
  2. The paper relies on automatic metrics and an LLM judge, without reporting human validation of answer correctness
  3. Figure 1 has very small fonts
Questions for authors
  1. Have the authors considered a sequence-based algorithm to learn storyline continuity, overlap, and splitting, reducing reliance on LLM-based judgments?

  2. Could the authors report performance under matched retrieval budgets, alongside each method’s best-performing configuration, to better establish the efficiency–accuracy trade-off?

4 comments

Thank you for your careful reading and helpful suggestions. We appreciate your recognition of the idea that one event can contribute to several independently evolving storylines. Your comments prompted us to add a matched-budget comparison, clarify what the memory design contributes, and improve the readability of Figure 1.

Q1. Could a sequence model replace the LLM judgments?

We agree that a sequence model is a promising direction. It could compare a new event with the ordered history of each candidate storyline, predict membership separately for each storyline, and learn when an existing storyline should split. The main challenge is supervision. The benchmarks we use provide questions, answers, and supporting evidence, but do not label which storylines an event should join or when a split is warranted. SUE-Mem can make these decisions with an off-the-shelf LLM, without collecting task-specific annotations or training a separate model. Learning these decisions from event sequences is a direction we would like to explore in future work. Sections 3.2 to 3.4 describe how the current system makes these decisions and updates memory.

Q2. How do the methods compare under matched budgets and in their main configurations?

We added a LoCoMo comparison of SUE-Mem, StructMem, and MRAgent under the same QA input-token caps of 2.5K, 5K, 10K, and 20K. SUE-Mem leads on J and BLEU-1 at all four caps, while StructMem leads on F1. At the tightest cap of 2.5K, SUE-Mem reaches 80.76 on J, ahead of StructMem at 72.32 and MRAgent at 55.95. This score exceeds both baselines’ J scores at 20K and is only 1.11 points below SUE-Mem’s own best score across the four caps. Section 4.9 reports all three metrics under matched caps.

W1. What does SUE-Mem contribute beyond prompting?

The LLM judges whether a new event belongs to a storyline. The memory design determines what happens after that judgment. SUE-Mem stores each event once, allows several storylines to reference it, and updates those storylines independently. For example, an award can close a competition storyline and satisfy a scholarship prerequisite. A later scholarship decision can extend the scholarship storyline without changing the competition storyline.

We tested this design by restricting each event to a single storyline. This produced the largest drop among the variants tested: Overall J fell from 81.56 to 73.98, and Multi-Hop J fell from 82.62 to 69.29. Section 3.3 describes the update rules, and Section 4.5 and Appendix D.1 report the ablation results.

W2. Why is there no separate human evaluation of answer correctness?

Thank you for suggesting an independent human evaluation. We randomly sampled 200 LoCoMo questions and asked two volunteers to judge the correctness of the generated answers, with each volunteer assessing a different set of 100 answers. The GPT-4o-mini judge agreed with the volunteers on 189 of the 200 answers, or 94.5%. Pearson’s r was 0.81 and 0.86 on the two sets, respectively. Appendix D.5 details the evaluation and disagreement analysis.

W3. Figure 1 readability

Thank you for pointing this out. We updated Figure 1 and expanded it to the full text width so the labels are easier to read. The revised figure appears at the start of Section 3.

Thank you for your response. I do understand your contribution however i will still keep my scores as the contribution is limited to the type of acceptances this conference had previously.

Thank you for taking the time to consider our response. We understand that your remaining concern is the scope of the contribution. SUE-Mem does not train a new model. It addresses a different research question: how should a memory system organize and update events when the same event belongs to several storylines that evolve independently? The LLM judges whether an event belongs to a storyline, while the memory system determines how shared events are stored and how each storyline is updated. Holding the backbone fixed lets us test that design choice directly. As reported in Section 4.5 and Appendix D.1, restricting each event to a single storyline lowers Multi-Hop J from 82.62 to 69.29. That comparison is the clearest evidence for the contribution we are making: a change to the memory system that improves multi-hop question answering without training a new model. Thank you again for your feedback, which helped us make both the contribution and its supporting evidence clearer.

Thank you very much again for your thoughtful consideration. We fully respect your judgment regarding the scope of our contribution.

We would like to re‑emphasize our core innovation: SUE‑Mem proposes a dedicated memory mechanism that enables a single sub‑event to be shared across, and dynamically update multiple independently evolving storylines. Different from existing memory frameworks that mostly bind one event to only one storyline, our work explores a new paradigm where events can participate in parallel story developments. To our knowledge, this specific multi‑storyline memory organization problem has not been systematically investigated in prior literature.

Our conclusion is solidly backed by three sets of empirical results:

(1) ablation study verifying the performance drop when disabling the cross‑storyline sharing design;

(2) matched‑budget comparison demonstrating competitive and balanced efficiency‑accuracy tradeoff;

(3) newly added human evaluation confirming the reliability of model outputs.

With these comprehensive supports, we sincerely hope you could kindly reconsider our work.

Reviewer_EEpqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SUE-Mem for long-term conversational memory management. The core idea is a pipeline that consists of extracting atomic events from conversations; retrieving candidate storylines based on the semantic similarities; using an LLM to determine whether an event belongs to a given storyline; allowing an event to join multiple storylines and periodically invoking an LLM to perform storyline splitting; and finally conducting retrieval using a combination of semantic, lexical, and entity/date/number anchors. The experimental results show that SUE-Mem achieves improvements on some Multi-Hop, Temporal, and Judge metrics on LoCoMo.

Critical strengths
  • The paper addresses a timely and important problem: organizing and retrieving long-term conversational memory to support cross-session reasoning.
  • The overlapping-storyline design is intuitive and well motivated, which allows an atomic event to participate in multiple independently evolving storylines.
  • The reported empirical results are promising, particularly on Multi-Hop and Temporal questions.
Critical weaknesses

My major concerns are regarding the paper's main contributions, the fairness of its experiments, and the scope of its conclusions. Overall, this work appears to combine several already common LLM memory engineering techniques rather than introducing a new memory representation, update mechanism, or retrieval algorithm.

  1. Limited novelty. I think the following works are highly related to this paper:
Related workOverlap with SUE-Mem
THEANINE: Towards Lifelong Dialogue Agents via Timeline-based Memory Management (2024/2025)Maintains memory units from long-term conversations, establishes temporal and causal relations, and organizes related events into timelines for response generation. Its overall pipeline already covers "event memory—relational organization—timeline retrieval—LLM generation."
Memory Matters More: Event-Centric Memory as a Logic Map for Agent Searching and Reasoning (CompassMem) (2026-01)Incrementally segments events, establishes explicit logical relations and an Event Graph, and performs goal-directed memory navigation based on this structure. It is particularly close to SUE-Mem's notion of "logic-guided event storylines."
ES-Mem: Event Segmentation-Based Memory for Long-Term Dialogue Agents (2026-01)Uses dynamic event segmentation and hierarchical memory, and locates episodic context based on event boundaries. SUE-Mem's atomic events and storyline splitting follow the same technical approach.
HiMem: Hierarchical Long-Term Memory for LLM Long-Horizon Agents (2026-01)Uses topic-aware event segmentation to construct Episode Memory and Note Memory, and performs semantic linking, hybrid retrieval, and conflict-aware reconsolidation. Its scope is even more comprehensive than that of SUE-Mem.
StructMem: Structured Memory for Long-Horizon Behavior in LLMs (2026-04-23)Uses event-level binding, cross-event connections, hierarchical memory, and periodic semantic consolidation, which are highly similar to SUE-Mem's storyline construction, summary regeneration, and splitting.
HingeMem: Boundary Guided Long-Term Memory with Query Adaptive Retrieval for Scalable Dialogues (2026-04-08)Uses boundary-triggered hyperedges to organize person/time/location/topic information and performs query-adaptive retrieval. It is very close to SUE-Mem's milestone-triggered splitting and hybrid retrieval.
SGMem: Sentence Graph Memory for Long-Term Conversational Agents (2025-09)Organizes conversations into a sentence-level graph, retains summaries, facts, and insights, and jointly retrieves the original dialogue and generative memory.
From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents (MemGAS) (2025-05)Constructs multi-granularity memory associations, uses clustering to associate old and new memories, and then performs selection through a router and LLM filtering.
TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025-02)Uses timeline summarization, infers event dates, and performs time-aware memory retrieval and temporal reasoning.
  1. Selectively reported results

Appendix C states that the LoCoMo adversarial category and the LongMemEval abstention task were excluded, without adequate justification. These categories are central to evaluating reliability because they test whether a system can recognize missing evidence and avoid retrieving irrelevant events. Since SUE-Mem relies on LLMs for event extraction, storyline assignment, summary integration, and answer generation, hallucination propagation and incorrect memory writes are also key risks.

Therefore, restricting the evaluation mainly to questions with retrievable answers does not adequately support the paper's broad claims about long-term memory reliability.

  1. Lack of baseline details

The paper does not sufficiently specify the baseline model versions, whether results were reproduced or copied, prompts, memory construction and retrieval budgets, context limits, and other hyper-params. The missing baseline metrics in Table 1 further obscure whether the comparisons were conducted under identical conditions.

The efficiency comparison is also incomplete. Table 3 reports only query-time tokens and latency using a reduced retrieval budget (K_s=1, K_h=3), while omitting the LLM-intensive writing costs of event extraction, per-candidate storyline judgments, summary regeneration, and splitting. Thus, lower query-time cost does not establish a lower end-to-end memory lifecycle cost. The paper should report total API calls, writing tokens, construction time, storage overhead, and how these costs scale with conversation history.

Questions for authors
  • What are the recall-based performances, such as retrieval recall, candidate recall, evidence coverage etc.? Merely F1/BLEU or LLM-judge scores are not sufficient for supporting your claims.

  • The paper uses the token cost and latency at query stage as the main efficiency indices, which oversees the cost from memory writing stage. What are the authors' thoughts on this? I have this concern because the writing cost per event per storyline candidate may be pretty high.

3 comments

Thank you for recognizing the value of overlapping storylines and for asking how well they recover evidence and what they cost to maintain. We address these questions with new measurements in the revised manuscript.

Q1. How much evidence does retrieval recover?

With 20 initial candidate storylines, SUE-Mem covers 98.22% of the annotated evidence turns. After selecting the child events to pass to the answer model, evidence coverage reaches 87.56% with 4 retained storylines, 93.13% with 8, and 96.02% with the default of 12. The default setting therefore retains nearly all the evidence coverage available in the candidate pool, with a gap of 2.20 percentage points. As you suggested, this measures retrieval directly rather than inferring it from answer scores. Section 4.7 discusses the results, and Appendix D.3 explains how we measure coverage using the source dialogue windows of the retrieved events.

Q2. What does memory construction cost?

You are right that the original efficiency comparison did not account for the work of assigning new events to storylines and updating their summaries. We now measure construction separately. On LoCoMo, SUE-Mem uses an average of 1.35 million input tokens and 0.09 million output tokens per conversation to build memory, taking 1,639.47 seconds on average. Its construction time is 18.9% shorter than StructMem’s. Across a complete LoCoMo run, including memory construction and all questions, SUE-Mem uses 16.37 million input tokens, compared with 22.58 million for MRAgent and 9.43 million for StructMem. Under this workload, SUE-Mem’s query-time savings more than offset its additional construction input relative to MRAgent, reducing total input tokens by 27.5%. Section 4.4 reports query-time efficiency, and Appendix D.4 reports construction and full-run input costs.

W1. What distinguishes SUE-Mem from prior work?

The overlap you point out in event extraction, logical links, and hybrid retrieval is real. Our contribution is how SUE-Mem maintains overlapping storylines as new events arrive. A single event can join several persistent storylines. The system decides membership separately for each storyline using its existing events, and each storyline can then evolve independently. For example, an award can close a competition storyline and satisfy a scholarship prerequisite. A later scholarship decision updates the scholarship storyline without changing the competition storyline.

This differs from navigating an event graph at query time to assemble evidence, as CompassMem does. It also differs from using dialogue boundaries to decide when to create a new memory segment. Our milestone check reassesses whether the events in an existing storyline still describe a coherent process. Removing multi-storyline assignment gives the largest performance drop among the variants we tested: Multi-Hop J falls from 82.62 to 69.29, and Overall J falls from 81.56 to 73.98. Sections 3.2 to 3.4 describe the update mechanism. Section 4.5 and Appendix D.1 report the ablation results.

W2. Why were adversarial questions excluded?

Most of the baseline results available for our main comparison did not report scores on LoCoMo adversarial questions. We initially focused the main comparison on the four question categories with broader baseline coverage. Your comment prompted us to evaluate the adversarial subset separately. SUE-Mem abstained on 402 of the 446 adversarial questions, yielding a semantic abstention rate of 90.13%. Appendix C.1 describes this new LoCoMo evaluation, while Appendix C.2 specifies the scope of our LongMemEval evaluation.

W3. Where do the baseline results come from, and how did we control the comparison?

We ran SUE-Mem locally and reproduced StructMem and MRAgent. The category-level F1 and BLEU-1 scores for A-MEM come from its published GPT-4o-mini results. The remaining baseline scores come from the LoCoMo comparison reported by Mem0. A dash denotes an unavailable score. Appendix C.1 identifies these sources and documents the retrieval settings used for both reproduced baselines.

Thank you for your time and effort in reviewing our paper. Your feedback is very valuable for improving this work.

Since the author-reviewer discussion period will end in about one day, we would appreciate it if you could take a look at our rebuttal when you have a moment and let us know whether your concerns have been addressed.

Thank you again for your time and suggestions.

Best regards, The authors

Thank you for your careful review of our work.

The author‑reviewer discussion window will close in a few hours. We have posted our rebuttal, in which we clarify the sources of baseline results and experimental comparison settings, together with supplementary experimental results.

We welcome you to review our response at your convenience. Please let us know if there are any remaining questions or comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.