acceptodds
Under review as a conference paper at ICLR 2027

ReplayMem: Long-Term Conversational Memory as Query-Conditioned Event Replay

Abstract

Long-term conversational memory enables large language models to use dialogue histories across sessions, track evolving user information, and provide personalized responses. Most existing methods transform conversations into memory representations that are constructed and continually maintained before future queries are known. Constructing these representations before queries are known can discard or distort source evidence, while repeated maintenance incurs LLM costs and can propagate earlier errors through subsequent updates. Based on the insight that query-relevant state can be constructed on demand from preserved source evidence, we propose ReplayMem, which formulates long-term conversational memory as query-conditioned event replay. ReplayMem stores original interactions as append-only source events and builds lightweight, source-linked retrieval views, which it uses to retrieve and expand relevant evidence into a query-specific event trajectory without continually maintaining a global memory state. In a single LLM invocation, the answering model replays this trajectory, explicitly constructs the query-relevant state, and generates the answer. This design preserves access to original evidence, avoids repeated state maintenance, and confines interpretation to the history relevant to the current query. Experiments on LongMemEval, PersonaMem-V2, and LoCoMo demonstrate state-of-the-art question-answering performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-term conversational memory, large language model agents, memory-augmented generation, query-conditioned retrieval, event replay, state reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5Zd7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ReplayMem preserves original conversations and combines dual-view indexing, semantic expansion, and temporal neighbors to retrieve evidence for answering questions in a single LLM call. Experiments suggest competitive performance across three conversational memory benchmarks.

Critical strengths

ReplayMem presents a clear and practical design for long-term conversational memory. Preserving original conversations keeps source evidence available, while dual-view indexing and iterative expansion help retrieve information distributed across the history. The answering stage requires only a single LLM call, without continually maintaining a global memory state. The reported reduction in memory-construction token costs also makes the approach potentially useful in practical applications.

Critical weaknesses

Major Concern 1: Evaluation consistency remains unclear. For memory baselines reported for LoCoMo and LongMemEval-S, scores are nearly identical to those in EverMemOS (Hu et al., 2026) under different evaluation settings ( averaged three judges for EverMemOS vs. GPT-4.1-mini judge for ReplayMem). The authors should clarify whether these results were reused or reevaluated under identical protocols. Without this clarification, the comparisons do not adequately support the SOTA claim.

Major Concern 2: Single-session preference understanding remains a weakness. Despite its strong overall performance, ReplayMem falls noticeably behind several baselines on SS-Pref in Table 1. The paper does not explain whether this gap arises from missing preference evidence during retrieval or inadequate interpretation during replay. A targeted failure analysis would clarify this limitation.

Major Concern 3: The benefit of state reconstruction is not isolated. The prompt in Appendix A.2.1 mainly asks the model to extract relevant information and reason step by step. Although Table 5 fixes the retrieved evidence, it lacks a standard step-by-step reasoning baseline. Thus, the improvement may reflect general reasoning instructions rather than a distinct state reconstruction mechanism.

Minor Concern 1: The supplied artifact is incomplete. Configuration files, installation metadata, and raw experimental logs are missing from the provided package. These artifacts should be provided for better reproducibility.

[1] Hu, C., Gao, X., Zhou, Z., et al. (2026). EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning. arXiv:2601.02163.

Questions for authors

See above. My questions mainly revolve around the major concerns. I understand reusing statistics due to consumption restrictions, but I do require clarifications on the experimental setting. I am willing to improve my score if most of my concerns are resolved.

Reviewer_LLTrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReplayMem, a long-term conversational memory method. It stores original dialogue chunks without later updates and uses an LLM to extract keyword and aspect tags for retrieval. At query time, it retrieves related chunks, expands them through the tags over several rounds, and asks the LLM to build the needed state before answering. It is tested on LongMemEval, PersonaMem-V2, and LoCoMo.

Critical strengths
  1. The paper is clearly written, and the train example in Figure 1 makes the problem with continual memory updates easy to understand.

  2. The method achieves the best overall results on three benchmarks, with a 7.4-point gain over EverMemOS on LongMemEval.

  3. The memory construction cost is low, since each chunk only needs one LLM call and stored memory is never updated.

Critical weaknesses
  1. The multi-hop retrieval is very similar to the graph retrieval in HippoRAG 2, which also links passages through extracted phrases. The authors should explain the difference and show what advantage their method has over HippoRAG 2.

  2. The paper does not report inference latency. At query time, the method runs several retrieval paths, multiple rounds of expansion, and reranking, so the latency is likely high. Only memory construction cost is compared.

  3. The motivation is that writing memory loses information and spreads errors. However, the gain on Knowledge Update is small. The largest gain is on Multi-Session, which is mainly a retrieval recall problem.

  4. The PersonaMem prompt seems tuned for this dataset, such as asking for the latest preference and whether it was withdrawn. The overall gain may come from this prompt. The author is recommend to compare retrieval performance with the baselines.

Questions for authors
  1. Could the authors analyze how the multi-hop retrieval differs from the graph retrieval in HippoRAG2, and explain what advantage their method has?

  2. Could the authors add experiments on inference cost and compare these with the baselines?

  3. Could the authors analyze why the gain on Knowledge Update is small? Could the authors also show which errors of the baselines come from memory writing and updating?

  4. Could the authors add experiments that run the baselines with the same PersonaMem prompt, or run ReplayMem with a generic prompt? Could the authors report retrieval performance compared with the baselines?

Reviewer_UjjoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReplayMem which formulates long term conversational memory as query conditioned event replay. ReplayMem stores original dialogue segments (the real original source text) as append only source events (not erasing or updating) and links each event to a keyword view and aspect view extracted by an LLM for retrieval. Given a query, ReplayMem retrieves anchor events through multiview retrieval and iteratively expands them through linked views under a token budget to build a query specific event trajectory. The trajectory contains only the original source text so the answering model reasons over raw dialogue rather than LLM generated memory representations. The answering model then constructs a query relevant state and generates the answer within a single LLM invocation.

Critical strengths

The gains are large on multi-session and temporal reasoning which require evidence spread across sessions.

Critical weaknesses
  1. What is the cause for difference between NaiveRAG and Replaymem for accuracy in Table 1? Every ablations in Table 5 is over 85% which eliminates the components of ReplayMem one by one. Maybe the improvement is not coming from view and expansion but reranker, retrieval config or prompt.

  2. How does explicit sqs_q differ from CoT prompting? The actual prompt in Figure 4 looks like a standard CoT instruction. Then which prompt is used for the w/o explicit sqs_q variant on Table 5? It is also unclear whether the baselines used the same CoT kind of answer prompt. Without this, ReplayMem shows 85.40% with reduced advances over comparisons in Table1 (Best comparison, EverMemOS, is 83%).

  3. Novelty compared to existing works is limited. The retrieval design is similar as graph-based RAG. HippoRAG[1] links LLM extracted keys to source passages, connects similar keys by embedding similarity, expands from query matched keys, returns the original passages, and handles new data by appending only. LightRAG[2]’s low-level and high-level keys are matched to the two level keyword and aspect views. A-MEM[3] also stores original content with LLM generated keywords and tags.

  4. ReplayMem moves interpretation to query time but authors don’t report the per-query cost (multi-round reranking, the input length of answering model and the extra output tokens for sqs_q). Since LongMemEval has only one question per history, reporting the query time cost will show the true total cost of ReplayMem.

[1] Gutiérrez et al, HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models

[2] Guo et al, LightRAG: Simple and Fast Retrieval-Augmented Generation

[3] Xu et al, A-MEM: Agentic Memory for LLM Agents

Questions for authors

Please see section Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.