acceptodds
Under review as a conference paper at ICLR 2027

Does Memory Construction Really Help the Long-Term Memory of AI Assistants?

Abstract

AI assistants need long-term memory to answer questions about users' past experiences. Existing methods reorganize these experiences into memory without considering the queries users will ask, but we find that their advantages diminish as evaluation spans more diverse benchmarks. In contrast, directly retrieving original records (Dense) remains competitive. To understand why, we keep the evidence and answers fixed and change only how questions are phrased. The resulting counterexamples show that rephrasing can turn a memory method's advantage over Dense into a disadvantage. This suggests that users' questions can play a decisive role in whether memory construction helps. Our theoretical analysis further shows that, under stated assumptions, no memory construction can guarantee a universal advantage over Dense without assumptions about the user's query distribution. This motivates Dynamic Retrieval Entrances for Agent Memory (DREAM), which preserves original records while adapting the routes used to retrieve them. As users ask questions, DREAM activates retrieval entrances that reflect their recent needs, allowing an initially retrieved record to lead to additional relevant evidence. Across four benchmarks, DREAM improves average Judge accuracy over Dense by 4.2 percentage points with Qwen3-VL-8B and 6.2 points with Gemma4-12B.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-term memory, AI Assistants, MLLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rHbeOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper asks whether structured long-term memory actually improves retrieval-based AI assistants. Across five memory benchmarks, dense retrieval over original records, called Dense, remains competitive as benchmark coverage broadens. The authors then rephrase questions while holding the memory source, evidence, and answers fixed. Geometry-guided counterexamples show that rephrasing alone can reverse a constructed-memory method’s ranking advantage over Dense. A no-free-lunch theorem under common-source ranking and target symmetry formalizes why no fixed constructor can dominate Dense over every admissible query/target distribution. DREAM keeps original records, generates candidate questions as retrieval entrances, and activates them using kernel-herding summaries and Hungarian matching over recent queries. Across four benchmarks, DREAM raises average Judge accuracy over Dense by 4.2 percentage points with Qwen3-VL-8B and 6.2 points with Gemma4-12B.

Critical strengths
  • The paper asks the right question: does restructuring interaction history into memory actually beat simply retrieving the original records? Framing Dense retrieval as a competitive baseline puts pressure on a design choice the field often assumes without testing.
  • The paper connects observation to theory to design. The benchmark comparison raises the question, the counterexample search and no-free-lunch analysis explain why query distribution can decide the answer, and DREAM turns that explanation into a mechanism that keeps raw records and adapts access routes.
  • DREAM's design directly targets the diagnosed failure mode. Rather than rewriting memory content, it generates candidate questions as retrieval entrances, activates them from recent queries through kernel herding and Hungarian matching, and follows them from dense hits to additional evidence.
  • The paper documents its evaluation protocol carefully. Appendices list the judge model and rubric, reader settings, local benchmark subsets, tokenizer budgets, and the bounded long-stream adaptations applied to RAPTOR, StructMem, and LightMem.
Critical weaknesses
  • The counterexamples are constructed, not observed. The search first optimizes query embeddings in Dense's favor, then uses LLM-driven rewriting to approach those targets. This shows that phrasing can reverse a retrieval advantage, but the reversal rate under search says little about how often ordinary user queries reverse. The paper also gives no human ratings of fluency, intent preservation, or naturalness for the generated questions.
  • The theorem is narrower than the surrounding prose implies. It assumes both systems rank the same set of items, but RAPTOR, Mem0, and MemoryBank rank summaries or tree nodes instead of raw records. Applying the theorem to those methods requires mapping their outputs back to the original sources, and the paper never shows that this mapping leaves their behavior intact. It also fails to demonstrate whether the theorem's favorable and unfavorable workloads resemble questions asked by real users. So the result rules out a universal guarantee in theory without saying much about the methods the paper evaluates.
  • The headline numbers comes from Qwen3-30B-A3B alone, with no human-agreement study and no second judge.
Questions for authors
  • What fraction of the 1,540 LoCoMo questions end in a successful reversal, and what fraction reverse under unoptimized paraphrases rather than the geometry-guided rewrites? For the reversal questions, how do human raters score fluency, intent preservation, and answer preservation?
  • Could you discuss how representative the chosen reversal questions are of real user scenarios? This would help readers better appreciate the practical implications of the theoretical findings.
4 comments
AuthorsOpenReview ↗

Response to Reviewer rHbe (1/2)

Thank you for recognizing the connection between our observations, theory, and DREAM. We address your questions about reversal frequency and real-user relevance, together with your concern about reliance on one judge.

Q1. How often does rewriting find Dense-favorable questions, and are the rewrites valid?

To assess how often rephrasing produces a question on which Dense retrieves the evidence more effectively, we use the same 1,540 LoCoMo questions as the paper's counterexample study. We compare two ways of rephrasing each question while keeping its intent and answer fixed: our geometry-guided search, which seeks wording that favors Dense, and ordinary paraphrasing, in which an LLM rewrites the question once without geometric guidance or retrieval-score feedback. This comparison tests whether Dense-favorable questions also arise without deliberately searching for them.

A search succeeds when the rewritten question preserves the original intent and answer, Dense ranks the supporting evidence above the memory method, and that evidence appears in Dense's top ten results. This measures success in finding a Dense-favorable question, regardless of which method led on the original wording. The table reports successful cases out of all 1,540 questions per method.

MethodGeometry-guided rewritingOrdinary paraphrasing
A-Mem1,143/1,540 (74.2%)403/1,540 (26.2%)
Mem01,086/1,540 (70.5%)451/1,540 (29.3%)
MemoryBank1,133/1,540 (73.6%)382/1,540 (24.8%)
RAPTOR1,206/1,540 (78.3%)503/1,540 (32.7%)

Geometry-guided rewriting finds Dense-favorable questions in 70.5–78.3% of cases, compared with 24.8–32.7% for ordinary paraphrasing. Thus, Dense-favorable questions arise under both procedures, and directed search finds them more frequently.

For human validation, two annotators independently assess 200 anchor–rewrite pairs, with 50 pairs per method and both successful and unsuccessful search outcomes represented. They see the source evidence and original answer, but not the method or search outcome. The table reports the number of pairs accepted by both annotators on each criterion.

Human-validation criterionBoth annotators acceptRate
Fluent and natural wording194/20097.0%
Original question intent preserved190/20095.0%
Original answer remains supported192/20096.0%
All three criteria satisfied184/20092.0%

The examples already provided in Appendix D, Figure A1, make this quality concrete. In panel (a), “What genre does Joanna's completed screenplay combine?” becomes “What genre fusion does Joanna's screenplay exemplify?” Both are fluent questions asking about the same screenplay genres, with the unchanged answer “It's a mix of drama and romance!” Yet the supporting-evidence ranks change from Dense/A-Mem = 4/2 to 2/6. This example illustrates how a natural reformulation can preserve the requested fact and answer while reversing the retrieval preference, complementing the aggregate human assessment above.

Q2. How representative are the failures, and what does the theory establish for practical memory methods?

The strongest evidence for relevance to ordinary questions comes from the original, unmodified LoCoMo questions in our appendix. Across three construction seeds, A-Mem improves overall F1 by 6.9–8.1 points on the same 1,527 evidence-eligible questions. Yet on questions where construction reduces query–support similarity by more than 0.05, the original-record baseline leads by 12.9–18.7 points:

Construction seedA-Mem gain on all questions (F1 points)Questions in the diagnostic subsetOriginal-record baseline gain on the subset (F1 points)
42+7.8173+12.9
43+6.9172+13.3
44+8.1159+18.7

Thus, the same construction improves overall performance while substantially reducing performance on identifiable groups of existing benchmark questions. Controlled rephrasing then isolates one cause: changing how the question is expressed can change which memory representation retrieves the evidence more effectively. Human validation in Q1 checks that these alternative expressions preserve the underlying information need.

Our theory connects these findings through a precise distinction: an average improvement does not establish improvement for every user-question distribution. The general proposition in Appendix C.1 applies directly to task outcomes, including those of summary-based methods. Once a question incurs a loss, concentrating demand on that question rules out universal dominance. The rank-based analysis in Appendix C.2 explains a mechanism for such losses when evidence rankings change. DREAM responds by using recent questions to allocate retrieval entrances to the user's observed needs, while preserving the original evidence.

AuthorsOpenReview ↗

Response to Reviewer rHbe (2/2)

Q3. Do the headline gains hold with an independent judge and human assessment?

We re-score the same saved Dense and DREAM answers with an independent judge. The gains retain approximately 90% of their original magnitude for both readers:

Answer assessmentDREAM gain with Qwen3-VL-8BDREAM gain with Gemma4-12B
Original judge+4.2 points+6.2 points
Independent judge+3.8 points+5.6 points

We additionally compare the original judge's binary correctness decisions with adjudicated human labels on 400 answers, balanced across four benchmarks, two readers, and Dense/DREAM. Agreement is 356/400 (89.0%). This evaluates judge–human agreement on answers separately from the rewrite-validity study in Q1.

Existing exact-match results provide further evidence without an LLM judge: DREAM improves Mem-Gallery by 2.5/3.2 points and MemEye multiple-choice accuracy by 5.8/8.7 points for Qwen/Gemma. Together, the independent judge, human agreement check, and exact-match metrics support gains beyond the original judge's scoring preferences.

Thank you for the author's reply and the experiment. I think the author has basically addressed my concerns. However, it is suggested that the specific methods and details of 'ordinary paraphrasing' be supplemented. I believe the theory of this article has certain novelty compared to previous studies, and the findings and entry points are also quite insightful. The experiment is also thorough. I will raise my original rating.

AuthorsOpenReview ↗

Thank you so much for your feedback!

Thank you for the care you put into reviewing our paper. We’re glad that our rebuttal and the additional experiment addressed your main concerns. As you suggested, we will revise the paper to explain how we performed “ordinary paraphrasing” and describe the procedure step by step.

Your encouraging words about the novelty of our theory, the insights from our findings, and the thoroughness of our experiments mean a great deal to us. We’re grateful for your thoughtful review!

Reviewer_ehb7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the memory construction approaches really benefit retrieval-augmented generation. It first compares different memory construction approaches and measures their performance compared to dense retrieval against the original records, establishing the fact that the dense retrieval against the original record performs well. It then proposes an approach called DREAM (Dynamic retrieval entrances for agent memory), which uses offline query generation to guide the retrieval path. Experiments on two open-source models showcased better performance of the proposed approach.

Critical strengths

The paper studies a timely question of whether constructing and organizing memory from past interactions through different approaches really benefit over retrieving from well-indexed raw records. It shows that dense retrieval against the original data record often performs really well for different user queries, and the performance of a given memory construction approach depends on how the queries are formulated (albeit unsurprising). The paper also tries to come up with a solution that can improve memory utilization and accuracy over previously proposed approaches, and experiments are run over five different benchmarks and different previous works.

Critical weaknesses

The paper is a bit challenging to understand. There are places that take a lot of attention to really parse what the paper is trying to say. And some parts feel less scientifically grounded. Although the paper narrows around the application of personalized AI assistants, I find, in many cases, it may just be a question of whether retrieval augmentation works with the right retrieved information. The paper should be more grounded in the intrinsic scientific question in the write-up.

There's no description of the benchmarks and their setups in the main text, presenting some difficulties in parsing the application scenarios and paper results.

The theoretical justification and the proposed theorems are only weakly related to the main piece of the algorithm. It is a bit unclear what the theorem brings to the table. The empirical results justify dense retrieval against the original record is good and dependent on how queries are formulated. Theory does not seem to be universally true according to practical scenarios. For example, if we have a perfect embedding model that can take different kinds of query writing, there might exist an overall better approach for document organization for retrieval.

The idea of query rewriting for the retrieval records is intuitive and interesting, but it was also seen in previous works, for example [1] is a missing reference which uses query proposal and speculation to organize the contexts and documents prepared for retrieval. Innovations should be more grounded in the related work.

[1] Li, Yanhong, David Yunis, David McAllester, and Jiawei Zhou. "Context-efficient retrieval with factual decomposition." In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 178-194. 2025.

Questions for authors

Could you, in plain language, explain the foundation of the proposed theorem? And how does it relate to the proposed algorithm for preparing a document to be retrieved?

What is the connection in this work to an AI assistant, not only in the motivation level?

How is long-term memory and time simulated, if there's any?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ehb7 (1/3)

Thank you for asking us to clarify the scientific question and the connection between agent memory (personalized AI assistants) and retrieval augmentation. Before addressing your three questions, we briefly clarify this background so that you can assess our work in its intended setting. In the standard RAG setting considered here, the system retrieves evidence from a static corpus to answer the current question, as in RAG, REALM, and Fusion-in-Decoder [1–3]. Agent memory extends this setting to a dynamic corpus, a memory archive whose contents are continually updated through interaction. The agent must maintain this archive to avoid losing important information and prevent outdated or conflicting memories from misleading later interactions.

During 2025–2026, a growing line of agent memory research, including A-Mem (NeurIPS 2025) [4], Mem0 (2025) [5], and LightMem (ICLR 2026) [6], explored how to construct better memories for this purpose. These works have received 1,017, 1,235, and 150 citations, respectively (OpenAlex, accessed November 16, 2026), reflecting substantial research attention to this direction. They retain retrieval as a means of accessing past information while emphasizing how to better preserve and use past experiences through memory organization, extraction, consolidation, or updating. This line of work motivates a shared research goal: finding a broadly effective memory-construction strategy without knowing the distribution of future user queries.

Our study grew out of following and evaluating these advances. However, extensive comparisons across five benchmarks revealed a recurring pattern: the benefits of these memory construction methods were less consistent across evaluation settings than individual benchmark results suggested. As benchmark coverage broadened, advantages over direct retrieval of original records often diminished. This pattern spans the six memory methods compared in Figure 1, rather than being confined to one implementation. These results led us to investigate the underlying cause: Does reorganizing a history into memory really improve on simply retrieving its original records?

Our paper makes four connected contributions: (1) a pilot evaluation across five benchmarks identifies the limited consistency of construction gains, (2) our theory explains the dependence on user demand under explicit assumptions, (3) geometry-guided question rephrasing constructs counterexamples while preserving the evidence and answer, and (4) DREAM introduces user demand into memory maintenance by adapting retrieval entrances. Now we address your questions about the theory, the assistant setting, and time below.

Q1. What is the foundation of your theory, and how does it motivate DREAM?

As described in Appendix C, our analysis of retrieval rankings over a fixed evidence set uses two assumptions:

(1) No user-demand prior: for a given question, the analysis allows any evidence source to be the target, rather than assuming in advance which sources the user will need.

(2) Earlier access is better: retrieving the required evidence at a higher rank receives a higher score, as in reciprocal rank.

Under this setting, our theory and propositions show that changing retrieval rankings creates gains for some needs and losses for others. A fixed memory construction therefore cannot guarantee an improvement over Dense across all user-question distributions. This conclusion does not depend on imperfect embeddings, including in your perfect-embedding example.

This identifies the missing ingredient for DREAM: user bias, the distribution of questions the user asks. DREAM estimates current demand from recent questions and activates the corresponding retrieval entrances while preserving original records. In essence, we construct a memory-maintenance scheme that changes with the user's needs.

AuthorsOpenReview ↗

Response to Reviewer ehb7 (2/3)

Q2. What is the connection to an AI assistant beyond the motivation?

As explained above, agent memory (personalized AI assistants) addresses a different task from single-question RAG over a static corpus. Standard document-based RAG typically accesses a static or infrequently updated knowledge base. In contrast, memory for personalized AI assistants must incorporate frequent updates from ongoing user–agent conversation sessions. Our experiments, theory, benchmarks, and method all address memory for personalized AI assistants:

① Our pilot experiments examine whether memory construction remains beneficial across a broad range of information needs of personalized AI assistants. Our theory identifies why a universal improvement requires assumptions about those needs.

② Our evaluation uses personalized AI assistant memory tasks from ATM, LoCoMo, Mem-Gallery, MemEye, and H2HMem, covering personal records, conversations, visual experiences, and information shared across people.

③ DREAM provides personalized AI assistants with a memory-maintenance mechanism that preserves original records and updates retrieval entrances using recent user questions.

Q3. How long are the memories, and how is time represented?

Our experiments cover memories spanning months to years. Our ATM evaluation includes 11,031 records dated from January 1, 2022 to July 21, 2025, a 3.55-year span, and evaluates all 1,044 questions in the loaded question set. LoCoMo contains 10 conversations, each spanning 19–35 sessions over 184–293 days, with 1,986 evaluation questions in total. The table reports timestamp spans in our evaluated data; ranges are across individual user or conversation streams.

BenchmarkMemory time spanEvaluated QA instances
ATMJan. 2022 to July 2025, 3.55 years1,044
LoCoMo184–293 days per conversation, 19–35 sessions1,986
Mem-Gallery70–216 days per stream1,711
MemEyeUp to 218 days per stream1,855
H2HMem21–356 days per stream1,969

The questions test recall of dated events, personal references, and information distributed across earlier sessions or records. For example, ATM requires locating personal evidence across images, videos, and emails; LoCoMo includes questions about event timing and relationships across sessions.

For the four multimodal benchmarks, we replay records and questions as an ordered stream. Memory persists between questions, and a query accesses only records that have arrived. DREAM updates its recent-query window at each question, with W=32 and γ=0.95 per query step. Thus, the benchmark timestamps supply the long-term history, while the replay supplies the sequential interaction and memory-update process. We will include these temporal spans and evaluation settings in the main text.

AuthorsOpenReview ↗

Response to Reviewer ehb7 (3/3)

What does DREAM add beyond Li et al.’s question speculation?

Thank you for pointing us to Li et al., Context-Efficient Retrieval with Factual Decomposition [7]. Their method first predicts questions that a document might answer, then extracts short facts guided by those questions. When a real question arrives, the system retrieves those facts. DREAM keeps the original records and uses the user’s recent questions to change how they are reached. An initial retrieval hit can lead to additional relevant records through the active entrances. The distinction is what adapts: Li et al. prepare retrievable facts from anticipated questions, whereas DREAM adapts access to original records using observed user questions.

To further address your concern, we compare DREAM with Li et al.’s method and two variants of our own retrieval mechanism. Direct candidate retrieval matches the current question to generated questions and returns their associated records. Window candidate retrieval also uses recent user questions, but returns records through candidate matching rather than DREAM’s entrance-based expansion. These two variants share DREAM’s candidate pool. The adapted Li et al. baseline retains its question generation, fact extraction, repeated sampling, and fact retrieval. All methods answer the same human-reviewed questions with a budget of 8,192 text tokens and 16 media inputs. The table reports Judge accuracy (%) for Qwen3-VL-8B/Gemma4-12B.

MethodATM Q/GMem-Gallery Q/GMemEye Q/GH2HMem Q/G
Direct candidate retrieval44.3/45.073.8/79.454.0/58.950.4/56.2
Window candidate retrieval45.0/45.774.4/80.054.8/59.551.0/56.9
Li et al., adapted factual retrieval44.7/45.474.1/79.754.4/59.250.7/56.6
DREAM46.6/47.476.3/81.956.7/61.353.0/58.8

DREAM outperforms all three comparators in all eight benchmark–reader combinations. Its average gains are 2.53/2.48 points over direct candidate retrieval, 1.85/1.83 over window candidate retrieval, and 2.18/2.13 over adapted factual retrieval. The two variants test whether generated questions and recent history alone explain the gain. The comparison with Li et al. tests whether preparing facts for retrieval achieves the same benefit. The results support the additional value of DREAM’s adaptive entrances to original evidence.

References

[1] Patrick Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.

[2] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval Augmented Language Model Pre-Training. ICML, 2020.

[3] Gautier Izacard and Edouard Grave. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. EACL, 2021.

[4] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-Mem: Agentic Memory for LLM Agents. NeurIPS, 2025.

[5] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413, 2025.

[6] Jizhan Fang et al. LightMem: Lightweight and Efficient Memory-Augmented Generation. ICLR, 2026.

[7] Yanhong Li, David Yunis, David McAllester, and Jiawei Zhou. Context-Efficient Retrieval with Factual Decomposition. NAACL-HLT, Volume 2: Short Papers, pp. 178–194, 2025.

Reviewer_NTcbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper questions whether constructed memories are always better than directly retrieving original records. The authors show that memory methods can lose their retrieval advantage when questions are rephrased, and provide a no-free-lunch analysis under certain assumptions.

Based on this, they propose DREAM, which keeps the original records and dynamically builds retrieval entrances based on recent user queries. Experiments on four benchmarks with two reader models show consistent improvements over Dense and several existing memory methods.

Critical strengths
  1. I like the motivation. Most memory papers focus on how to summarize or organize past information, while this paper looks at whether the constructed memory is actually easier to retrieve. The question-rephrasing experiments are interesting, even though I have some reservations about their interpretation.

  2. The experiments are fairly comprehensive. Results are consistent across four benchmarks and two readers. I especially appreciate Table 3, which compares against candidate-question retrieval and neighbor expansion under the same evidence budget. The ablations also help explain why the different components of DREAM are useful.

Critical weaknesses
  1. I think the paper somewhat overstates the limitations of memory construction. The counterexamples are explicitly optimized to favor Dense, so they don't tell us how frequently such failures happen for real user questions. The theoretical result also assumes that both methods rank the same sources and that any source can be the correct target for a given query. This is quite different from many actual memory construction methods that compress or reorganize information. I see the theorem as a useful observation, but not a particularly strong argument against memory construction in general.

  2. I'm not fully convinced that the experiments demonstrate adaptation to changing user needs. DREAM relies heavily on recent query history, but the benchmarks do not really evaluate realistic changes in query distributions over time. Also, the method generates up to 16 candidate questions per record and maintains additional retrieval links. The storage and maintenance overhead could become significant for long-term assistants, but there is no direct cost or latency comparison.

Questions for authors

In Figure 1(c,d), the results for smaller benchmark subsets are compared against Dense's fixed five-benchmark average, rather than Dense on the same subset. Would the conclusion still hold with matched benchmark subsets?

Have the authors tested DREAM under explicit query distribution shifts? For example, when a user's interests change substantially, how quickly does the method adapt?

What is the actual construction and update cost of DREAM compared with Dense and other memory methods? I'm particularly interested in how this scales with the number of stored records.

2 comments
AuthorsOpenReview ↗

Response to Reviewer NTcb (1/2)

Thank you for the positive assessment and the questions on matched comparisons, adaptation, and cost. We address each question below.

Q1. Does the conclusion hold when Dense is evaluated on the same benchmark subsets?

Yes. We report both methods on each of the 31 nonempty benchmark subsets. Across the full five benchmarks, Dense matches or exceeds all seven Figure 1 comparators in both reader panels. The table contrasts each comparator’s largest matched single-benchmark difference with its matched five-benchmark result (comparator minus Dense, percentage points; Q/G denotes the two reader panels).

ComparatorLargest single-benchmark difference Q/GMatched five benchmarks Q/G
A-Mem−3.04 / −3.71−4.87 / −6.29
Mem0+11.78 / +11.78−0.03 / −2.21
HippoRAG 2+4.23 / +4.23−3.05 / −5.38
BM25+2.32 / +2.32−1.76 / −2.91
CoM+0.96 / 0.00−1.53 / −2.52
RAPTOR+0.37 / +0.22−1.63 / −1.84
StructMem+1.36 / +1.36−6.90 / −10.04

For example, Mem0’s best single-benchmark advantage is +11.78 points, but its five-benchmark difference is −0.03/−2.21 points, −0.03 is effectively a tie. Thus, the observation that individual-benchmark gains need not persist at broader coverage holds with matched subsets. As in Figure 1, both panels use the same Qwen3-30B reader for LoCoMo. These paired comparisons establish Dense’s competitiveness.

Q2. How quickly does DREAM adapt to explicit changes in user interests?

With W=32 and γ=0.95, new-interest queries account for 23.0%, 41.7%, 69.4%, and 87.8% of the history weight after 4, 8, 16, and 24 queries. Adaptation therefore starts before the history window is replaced. In the shift evaluation below, DREAM already exceeds current-query-only activation in the first eight-query block and strengthens this advantage over the next two blocks.

We test two modes of explicit changes in user interests, A→B and A→B→A, on ATM and Mem-Gallery with two readers (Qwen3-VL-8B and Gemma4-12B). A and B are distinct event or topic groups within the same user's archive: A queries one event and its associated people or activities; B switches to another event with different associated entities. Each phase contains 32 questions. The return phase uses new questions about A, not repeated questions. We average over user streams and three within-topic query-order seeds. “Frozen” holds the entrances fixed at their configuration immediately before each switch; “Current” activates entrances from the current question alone.

The first table reports Judge accuracy (%) after A→B. Current and Frozen are evaluated over the same query blocks, with their three block scores listed in order. All methods use the same questions within each block.

Dataset / readerCurrent, queries 1–8 / 9–16 / 17–24Frozen, queries 1–8 / 9–16 / 17–24DREAM, queries 1–8DREAM, queries 9–16DREAM, queries 17–24
ATM / Qwen42.1 / 41.4 / 41.940.9 / 39.8 / 40.642.843.444.4
ATM / Gemma40.2 / 40.9 / 40.738.8 / 39.7 / 39.040.842.743.0
Mem-Gallery / Qwen69.1 / 68.3 / 69.066.9 / 66.6 / 66.570.371.773.4
Mem-Gallery / Gemma74.0 / 74.8 / 74.471.4 / 72.5 / 71.575.579.179.9

Across the four dataset–reader pairs, DREAM gains 0.6–1.5 points over Current in queries 1–8, 1.8–4.3 points in queries 9–16, and 2.3–5.5 points in queries 17–24. Thus, 16–24 queries describe the approach to the sustained gain, not a waiting period before adaptation helps.

The second table reports the return B→A. Frozen now retains the B-adapted configuration from immediately before the return, testing whether updating entrances is needed when a previous interest reappears.

Dataset / readerCurrent, queries 1–8 / 9–16Frozen, queries 1–8 / 9–16DREAM, queries 1–8DREAM, queries 9–16
ATM / Qwen42.4 / 41.841.5 / 40.543.043.7
ATM / Gemma40.3 / 40.839.1 / 39.840.942.5
Mem-Gallery / Qwen68.5 / 69.266.9 / 67.169.672.5
Mem-Gallery / Gemma74.9 / 74.272.5 / 72.476.378.3

After returning to A, DREAM gains 0.6–1.4 points over Current in queries 1–8 and 1.7–4.1 points in queries 9–16, without repeating earlier questions or generating new candidates. Together, the two shifts show that recent-query activation responds to changing interests rather than benefiting only a fixed query distribution.

AuthorsOpenReview ↗

Response to Reviewer NTcb (2/2)

Q3. What are DREAM’s construction and update costs compared with Dense and other memory methods, and how do they scale?

At 10,000 records, DREAM takes 0.63 hours to construct. A-Mem, Mem0, and RAPTOR take 13.69, 2.41, and 4.05 hours, respectively (2 L40 GPUs).

DREAM adds 80 ± 12 ms per query over Dense. This is the mean ± sample standard deviation of paired run-level latency differences across 24 runs: four benchmarks (ATM, Mem-Gallery, MemEye, and H2HMem), two readers (Qwen3-VL-8B and Gemma4-12B), and three query-order seeds, each with a 10,000-record archive. Each run evaluates 100 questions, giving 2,400 timed query executions per method. Mean end-to-end latency is 8.02 ± 0.64 seconds for Dense and 8.10 ± 0.64 seconds for DREAM, with the same run-level aggregation. The ratio of these means gives a 1.0% latency increase.

This overhead supports adaptation without repeated construction. DREAM generates and caches up to 16 candidate questions per incoming record, and interest updates select existing candidates and require zero additional LLM generation calls. In the main evaluation, DREAM improves average Judge accuracy over Dense by 4.2/6.2 points with Qwen3-VL-8B/Gemma4-12B.

For 1,000/10,000/100,000 records, DREAM’s construction time is 0.06/0.63/13.83 hours and its resident retrieval-data footprint is 0.13/1.25/12.50 GiB. The footprint includes FP32 vectors, candidate text, and entrance indexes, excluding model weights and raw-media files. At 100,000 records, candidate vectors account for 9.155 GiB, calculated as 16 candidates × 1,536 dimensions × 4 bytes per record. Thus, storage scales linearly with archive size, and the initial construction is reused across subsequent interest changes.

Additional concern: Are the findings limited to optimized counterexamples or the theorem’s assumptions?

The appendix also tests original, unmodified LoCoMo questions. Across three construction seeds, A-Mem leads by 7.8, 6.9, and 8.1 F1 points on the same 1,527 evidence-eligible questions. Yet on the subsets where construction reduces query–support similarity by more than 0.05, the original-record baseline leads by 12.9, 13.3, and 18.7 points. These subsets contain 173, 172, and 159 questions (10.4–11.3% of the eligible set). Thus, substantial conditional disadvantages also occur without optimized rewriting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.