acceptodds
Under review as a conference paper at ICLR 2027

DyMem: Query-Adaptive Relational Memory for Long-Term Conversation

Abstract

Long-term conversational agents must reason over information accumulated across many interactions, where answering a question often requires connecting distributed memories through temporal, topical, or entity-level relations. Existing flat retrieval methods rank memories independently and may overlook evidence that is weakly aligned with the query but connected to relevant memories through meaningful relations. Graph-based memory systems make such dependencies explicit, yet a persistent graph defines only a space of possible connections, while fixed or query-agnostic traversal can follow irrelevant relational paths and introduce unnecessary context. We propose DyMem, a query-adaptive relational memory framework that conditions retrieval not only on memory-node relevance, but also on the semantic meaning of the relations connecting memories. DyMem constructs a persistent heterogeneous memory graph with explicit relation descriptions and dynamically activates query-relevant relations to form a query-specific evidence subgraph. The resulting candidates are further pruned under a context budget and mapped back to the original dialogue turns for source-grounded answer generation. Experiments on LoCoMo and LongMemEval across multiple language model backbones show that DyMem consistently improves long-term conversational question answering over representative memory baselines, with particularly strong gains on multi-hop and temporal queries while maintaining a compact retrieval context. These results highlight the importance of treating relations themselves as query-dependent retrieval signals in long-term conversational memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent Memory, Large Language Model Agents, Memory Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YRZCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates query-adaptive relational evidence selection for long-term conversational memory and proposes DyMem, which activates typed edges via query relation alignment and grounds evidence back to verbatim dialogue turns. Gains on multi-hop and temporal QA are promising, with much lower context cost.

Critical strengths

1.The framework of DyMem is well-developed and highly targeted: DyMem defines the core challenge of long-term memory as not merely how to represent relations, but how to determine which relations themselves are relevant to a specific information need. Based on this, DyMem designs a retrieval method that balances graph size and relevance. The method is reasonable and is not a simple stacking of modules.

2.Experiments are thorough: Results across multiple base models and benchmarks show that DyMem achieves competitive performance while maintaining moderate retrieval and construction overhead.

Critical weaknesses

Memory extraction is performed by an LLM, but the paper lacks an analysis of extraction quality and its impact on retrieval. Adding a qualitative analysis of extraction quality and a quantitative analysis with a different extraction model would better demonstrate the stability of the method.

There are some writing issues. The REPRODUCIBILITY STATEMENT contains content unrelated to the paper, such as Navier-Stokes, Compressible Navier-Stokes, Shallow Water Equations, and Diffusion-Reaction. In addition, Table 12 indicates that the original logs need to be checked; the authors are advised to proofread the paper carefully.

The results show a significant increase in retrieval time, but the paper does not analyze this phenomenon. The authors are advised to add a corresponding analysis.

Questions for authors

see above

2 comments

W1:

To investigate the impact of the Extractor, we conducted experiments using Qwen2.5-3B and GPT-4o-mini. To the best of our knowledge, GPT-4o-mini has substantially stronger capabilities than Qwen2.5-3B.

In these experiments, we only replaced the Extractor model while keeping all subsequent components and settings unchanged.

The experimental results are presented as follows:

ExtractorAnswer modelMulti-hopTemporalOpen-domainSingle-hopOverall
GPT-4o-miniQwen2.5-3B22.1517.7213.5941.6531.34
Qwen2.5-3BGPT-4o-mini40.3050.6828.7959.4452.20

As shown, the capability of the Extractor does influence memory performance to some extent. Replacing the Extractor with either GPT-4o-mini or Qwen2.5-3B leads to a corresponding increase or decrease in performance. However, the magnitude of this variation is limited (approximately ±1%), indicating that our method does not heavily rely on the capability of the Extractor.

W2:

Due to an asynchronous issue during the template merging process, several typos were introduced in the Reproducibility Statement. We have corrected all of them in the latest revised manuscript. We are also carefully checking and correcting any other potential typos throughout the paper. We would greatly appreciate it if you could let us know if you identify any further typos.

W3:

To provide a clearer analysis of the time consumption during retrieval, we decompose DyMem's query-time retrieval process into six stages and measure the execution time of each stage separately. The stages and their corresponding operations are as follows:

  • Query Encoding: Query and context embedding.
  • Start Point Retrieval: Memory and topic retrieval.
  • Relation Activation: Semantic matching between the query and relation descriptions.
  • Graph Traversal: Path-based search to retrieve the corresponding memory nodes.
  • Evidence Selection: Computation of evidence coverage, redundancy, and length.
  • Source Grounding: Localization of relevant information in the original conversations.

The analysis is presented as follows:

StageTime (ms)Proportion (%)
Query Encoding10.88 ± 1.650.61
Seed Retrieval24.82 ± 8.521.40
Relation Activation70.48 ± 30.643.98
Graph Traversal14.21 ± 4.070.80
Evidence Selection1647.83 ± 1029.9293.05
Source Grounding2.78 ± 0.810.16
End-to-end retrieval1771.00 ± 1060.00100.00

The main reason is that computing scores for the retrieved evidence introduces additional computational overhead.

Thank you for your response.

Reviewer_WP7WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces DyMem, a query-adaptive relational memory framework for long conversations, where similarity search can miss indirect evidence and fixed graph traversal can retrieve redundant information. It organizes past conversations into a memory graph with descriptions of the relations, then activates query-relevant relations to connect scattered evidence. Within a context budget, it balances relevance, coverage, and redundancy when selecting memories and retrieves the original dialogue snippets to preserve factual details. Experiments on LoCoMo and LongMemEval show gains over several memory baselines, with ablations showing that both dynamic relation selection and access to the original dialogue contribute to these gains.

Critical strengths

The paper is very well written and easy to follow. The motivation is clear, Figure 1 explains the main idea well, and Figure 2 makes the workflow easy to understand. The tables are detailed, the ablations are thorough, and the results come with substantial analysis. Overall, the presentation feels clear and complete.

Critical weaknesses

I found the evidence-efficiency results a little puzzling: performance initially improves as more context is added, but then starts to decline. I would have liked a clearer explanation of why adding more evidence eventually hurts performance, and what this means for scaling the method with a larger inference budget.

The gains also seem to peak at around 3K tokens, which is a fairly small context budget for an LLM. This leaves me unsure whether DyMem can make effective use of substantially longer contexts, or whether the observed behavior points to a limitation of the method.

Questions for authors

What causes performance to decline as the context budget increases? Have you explored substantially larger budgets beyond 3K tokens? More broadly, can DyMem benefit from additional test-time compute, either through more retrieved context or through other forms of inference?

1 comment

We sincerely thank the reviewer for this insightful suggestion.

This is indeed an interesting question.

In fact, more retrieved memories do not necessarily lead to better performance; what matters is the precision of memory retrieval. Once sufficient information has been retrieved to answer a query, forcing further retrieval may introduce additional noise rather than provide useful evidence.

To illustrate this issue more clearly, we conducted an additional analysis of the retrieval results and introduced two additional metrics:

  • Evidence Precision, measures the average precision of the retrieved memory nodes with respect to the Gold Evidence.
  • Redundancy Ratio, measures the average similarity between each newly retrieved node and the previously retrieved nodes.

The results are presented as follows:

Token budgetEvidence precision (%) ↑Redundancy ratio (%) ↓
1K24.40.00
2K31.10.49
3K32.91.42
5K26.13.66
8K17.96.62
13K17.010.45
Reviewer_Kc6bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work investigates long-term conversational memory: answering questions whose evidence is spread across many sessions and linked only through temporal, topical, or entity-level relations. To address it, the authors proposed DyMem, a heterogeneous memory graph whose edges carry explicit textual relation descriptions. Different from existing work that focuses on flat retrieval ranking memories independently, or graph memory traversed in a fixed, query-agnostic way, DyMem activates edges per query by matching the query against the relation descriptions, then runs bounded traversal and utility-driven selection under a context budget, answering from the verbatim dialogue turns behind the selected nodes. Experimental results verify its effectiveness and superiority: across LoCoMo and LongMemEval on six LLM backbones, DyMem reports consistent gains over flat and graph-based baselines — e.g., Temporal F1 50.00 vs. A-Mem's 24.42 on GPT-4o-mini — while retrieving 2,157.9 context tokens per question vs. A-Mem's 18,679.0.

Critical strengths
  • A genuinely differentiated retrieval signal. Traversal is conditioned on the semantics of relations, not just node similarity: edges store typed textual descriptions, and query-time activation scales the edge weight by query–relation alignment — a clear step beyond static graph expansion.
  • Source grounding is the load-bearing component, and the ablation proves it. Removing source-preserving reading collapses Avg F1 from 30.66 to 9.58 on Qwen2.5-3B — by far the largest drop of any ablation — confirming the design's core premise (Table 3)
Critical weaknesses
  • The paper presentation is bad, a lot of empty space while the captions are pretty close to tables, Figure 4's resolution is pretty low, hard to read;
  • DyMem's Overall scores (LLM 0.672, F1 0.438) are the unweighted mean of its six category scores, while Full-Context's and A-Mem's are per-question aggregates; under that convention DyMem's LLM score is ≈0.619, essentially tied with Nemori's 0.622. Its Overall also exceeds the Full-Context reference (0.565; F1 0.113), contradicting its description as an "upper-bound reference" (Table 2).
  • The encoder is all-MiniLM-L6-v2 "post-trained for our setting," but the post-training data and objective are never given — a fairness and leakage confound — and no baseline's embedder is stated. LongMemEval coverage is thin (only A-Mem and Nemori, one backbone, several categories lost), and the LLM judge behind the headline metric is never identified despite a promise to report it. Main results also report no variance, and SimpleMem's LLaMA3.2-1B row is unexplained dashes.
Questions for authors
  • Which aggregation defines "Avg" in Table 1 and "Overall" in Table 2? Please re-report every row under a single convention: under the unweighted one, DyMem's Qwen2.5-1.5B Avg F1 (24.86) falls below SimpleMem's (25.23).
  • If query-conditioned traversal is the mechanism behind the temporal gains, why does removing graph traversal raise Temporal F1 by ~9 points on Qwen2.5-3B (25.84 vs. 16.94, Table 3)?
  • Were the category-specific answer prompts in Appendix D used for all methods, or only for DyMem? The temporal prompt teaches date conversion and the multi-hop prompt instructs the model not to answer "Not mentioned" when inference is supported — how much of the Temporal/Multi-Hop advantage survives a shared neutral prompt?
  • How does the author see AssoMem compared with this work?
3 comments

Q4:

The most fundamental distinction between the two approaches is as follows:

AssoMem's adaptivity primarily operates at the level of multi-signal scoring and ranking of memory nodes.

DyMem's adaptivity primarily operates at the level of selecting relationships between memory nodes and determining graph search paths.

In other words, AssoMem focuses on which memories are more important, while DyMem focuses on which relationships should be followed to retrieve the required evidence.

W1:

We have strictly followed the official formatting template. Any potential formatting issues are due to compromises made to accommodate the page limit, and we will revise the formatting accordingly.

However, we would appreciate it if the reviewer could specify which figure caption and table appear to run together. We have re-examined the manuscript and confirmed that all figures and tables are presented separately in the main text.

Regarding the clarity of Fig. 4, we will replace it with a vector graphic to further improve its visual quality. Nevertheless, based on our inspection of the current version, the figure remains sufficiently clear and does not hinder readability.

W2/W3/Q1:

Due to an asynchronous issue during template merging, several typos were introduced in the Table , the Method section, and the Reproducibility Statement. We have corrected all of these issues in the latest revised manuscript. We have also made every effort to check and fix other potential typos throughout the paper. We would greatly appreciate it if the reviewer could let us know of any additional issues they may identify.

Regarding SimpleMem, the use of a dash indicates that it is not applicable to this particular backbone model. Please refer to its open-source implementation and original paper for further details.

Regarding the experimental evaluation protocols, we strictly follow the official releases of LoCoMo and LongMemEval, which are widely adopted benchmarks in the LLM memory research community. To ensure a fair comparison, we select the same evaluation categories as Nemori. For LLM-based evaluation metrics, consistent with other existing memory-related works, we use GPT-4o-mini as the evaluator.

Furthermore, to provide a more comprehensive comparison, we have included the full weighted evaluation version in the appendix of the latest revision, along with a detailed explanation of the experimental protocols and evaluation criteria.

Q2:

We divide the 321 Temporal queries in LoCoMo into two groups based on the number of sessions involved in their Gold Evidence:

  • Single-session Temporal: The answer evidence involves only one session, requiring direct retrieval of the target event and its corresponding temporal information.
  • Cross-session Temporal: The answer evidence spans two or more sessions, requiring the association of multiple events and their temporal relationships.

The results are presented as follows:

SubsetQuestionsFull F1 (%)w/o Graph Traversal F1 (%)Δ (pp)
All temporal32116.9425.84-8.9
Single-session29316.7427.82-11.08
Cross-session2819.055.12+13.93

Because of graph traversal, our method places greater emphasis on searching through dynamically selected relationships between memory nodes, rather than directly retrieving individual memory nodes.

Q3: First, existing LLM memory systems, such as A-Mem, Mem0, and SimpleMem, all design and adapt their prompts according to their respective memory modeling strategies and characteristics.

Second, we hope the reviewer can appreciate that LLM memory is an inherently complex modeling problem. We must deal with massive conversational histories that far exceed the context window of LLMs. This requires continuous investigation into how to construct, retrieve, and effectively utilize memory. We believe that researchers working in this area would generally agree that the memory problem cannot be adequately addressed through simple prompt engineering alone.

Finally, all results reported in our paper are obtained using the best-performing configurations of our method.

Thank you for bringing this paper to our attention. We will include it in our references.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.