- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper asks whether structured long-term memory actually improves retrieval-based AI assistants. Across five memory benchmarks, dense retrieval over original records, called Dense, remains competitive as benchmark coverage broadens. The authors then rephrase questions while holding the memory source, evidence, and answers fixed. Geometry-guided counterexamples show that rephrasing alone can reverse a constructed-memory method’s ranking advantage over Dense. A no-free-lunch theorem under common-source ranking and target symmetry formalizes why no fixed constructor can dominate Dense over every admissible query/target distribution. DREAM keeps original records, generates candidate questions as retrieval entrances, and activates them using kernel-herding summaries and Hungarian matching over recent queries. Across four benchmarks, DREAM raises average Judge accuracy over Dense by 4.2 percentage points with Qwen3-VL-8B and 6.2 points with Gemma4-12B.
Critical strengths
- The paper asks the right question: does restructuring interaction history into memory actually beat simply retrieving the original records? Framing Dense retrieval as a competitive baseline puts pressure on a design choice the field often assumes without testing.
- The paper connects observation to theory to design. The benchmark comparison raises the question, the counterexample search and no-free-lunch analysis explain why query distribution can decide the answer, and DREAM turns that explanation into a mechanism that keeps raw records and adapts access routes.
- DREAM's design directly targets the diagnosed failure mode. Rather than rewriting memory content, it generates candidate questions as retrieval entrances, activates them from recent queries through kernel herding and Hungarian matching, and follows them from dense hits to additional evidence.
- The paper documents its evaluation protocol carefully. Appendices list the judge model and rubric, reader settings, local benchmark subsets, tokenizer budgets, and the bounded long-stream adaptations applied to RAPTOR, StructMem, and LightMem.
Critical weaknesses
- The counterexamples are constructed, not observed. The search first optimizes query embeddings in Dense's favor, then uses LLM-driven rewriting to approach those targets. This shows that phrasing can reverse a retrieval advantage, but the reversal rate under search says little about how often ordinary user queries reverse. The paper also gives no human ratings of fluency, intent preservation, or naturalness for the generated questions.
- The theorem is narrower than the surrounding prose implies. It assumes both systems rank the same set of items, but RAPTOR, Mem0, and MemoryBank rank summaries or tree nodes instead of raw records. Applying the theorem to those methods requires mapping their outputs back to the original sources, and the paper never shows that this mapping leaves their behavior intact. It also fails to demonstrate whether the theorem's favorable and unfavorable workloads resemble questions asked by real users. So the result rules out a universal guarantee in theory without saying much about the methods the paper evaluates.
- The headline numbers comes from Qwen3-30B-A3B alone, with no human-agreement study and no second judge.
Questions for authors
- What fraction of the 1,540 LoCoMo questions end in a successful reversal, and what fraction reverse under unoptimized paraphrases rather than the geometry-guided rewrites? For the reversal questions, how do human raters score fluency, intent preservation, and answer preservation?
- Could you discuss how representative the chosen reversal questions are of real user scenarios? This would help readers better appreciate the practical implications of the theoretical findings.
4 comments
Response to Reviewer rHbe (2/2)
Q3. Do the headline gains hold with an independent judge and human assessment?
We re-score the same saved Dense and DREAM answers with an independent judge. The gains retain approximately 90% of their original magnitude for both readers:
| Answer assessment | DREAM gain with Qwen3-VL-8B | DREAM gain with Gemma4-12B |
|---|---|---|
| Original judge | +4.2 points | +6.2 points |
| Independent judge | +3.8 points | +5.6 points |
We additionally compare the original judge's binary correctness decisions with adjudicated human labels on 400 answers, balanced across four benchmarks, two readers, and Dense/DREAM. Agreement is 356/400 (89.0%). This evaluates judge–human agreement on answers separately from the rewrite-validity study in Q1.
Existing exact-match results provide further evidence without an LLM judge: DREAM improves Mem-Gallery by 2.5/3.2 points and MemEye multiple-choice accuracy by 5.8/8.7 points for Qwen/Gemma. Together, the independent judge, human agreement check, and exact-match metrics support gains beyond the original judge's scoring preferences.
Thank you for the author's reply and the experiment. I think the author has basically addressed my concerns. However, it is suggested that the specific methods and details of 'ordinary paraphrasing' be supplemented. I believe the theory of this article has certain novelty compared to previous studies, and the findings and entry points are also quite insightful. The experiment is also thorough. I will raise my original rating.
Thank you so much for your feedback!
Thank you for the care you put into reviewing our paper. We’re glad that our rebuttal and the additional experiment addressed your main concerns. As you suggested, we will revise the paper to explain how we performed “ordinary paraphrasing” and describe the procedure step by step.
Your encouraging words about the novelty of our theory, the insights from our findings, and the thoroughness of our experiments mean a great deal to us. We’re grateful for your thoughtful review!
Response to Reviewer rHbe (1/2)
Thank you for recognizing the connection between our observations, theory, and DREAM. We address your questions about reversal frequency and real-user relevance, together with your concern about reliance on one judge.
Q1. How often does rewriting find Dense-favorable questions, and are the rewrites valid?
To assess how often rephrasing produces a question on which Dense retrieves the evidence more effectively, we use the same 1,540 LoCoMo questions as the paper's counterexample study. We compare two ways of rephrasing each question while keeping its intent and answer fixed: our geometry-guided search, which seeks wording that favors Dense, and ordinary paraphrasing, in which an LLM rewrites the question once without geometric guidance or retrieval-score feedback. This comparison tests whether Dense-favorable questions also arise without deliberately searching for them.
A search succeeds when the rewritten question preserves the original intent and answer, Dense ranks the supporting evidence above the memory method, and that evidence appears in Dense's top ten results. This measures success in finding a Dense-favorable question, regardless of which method led on the original wording. The table reports successful cases out of all 1,540 questions per method.
Geometry-guided rewriting finds Dense-favorable questions in 70.5–78.3% of cases, compared with 24.8–32.7% for ordinary paraphrasing. Thus, Dense-favorable questions arise under both procedures, and directed search finds them more frequently.
For human validation, two annotators independently assess 200 anchor–rewrite pairs, with 50 pairs per method and both successful and unsuccessful search outcomes represented. They see the source evidence and original answer, but not the method or search outcome. The table reports the number of pairs accepted by both annotators on each criterion.
The examples already provided in Appendix D, Figure A1, make this quality concrete. In panel (a), “What genre does Joanna's completed screenplay combine?” becomes “What genre fusion does Joanna's screenplay exemplify?” Both are fluent questions asking about the same screenplay genres, with the unchanged answer “It's a mix of drama and romance!” Yet the supporting-evidence ranks change from Dense/A-Mem = 4/2 to 2/6. This example illustrates how a natural reformulation can preserve the requested fact and answer while reversing the retrieval preference, complementing the aggregate human assessment above.
Q2. How representative are the failures, and what does the theory establish for practical memory methods?
The strongest evidence for relevance to ordinary questions comes from the original, unmodified LoCoMo questions in our appendix. Across three construction seeds, A-Mem improves overall F1 by 6.9–8.1 points on the same 1,527 evidence-eligible questions. Yet on questions where construction reduces query–support similarity by more than 0.05, the original-record baseline leads by 12.9–18.7 points:
Thus, the same construction improves overall performance while substantially reducing performance on identifiable groups of existing benchmark questions. Controlled rephrasing then isolates one cause: changing how the question is expressed can change which memory representation retrieves the evidence more effectively. Human validation in Q1 checks that these alternative expressions preserve the underlying information need.
Our theory connects these findings through a precise distinction: an average improvement does not establish improvement for every user-question distribution. The general proposition in Appendix C.1 applies directly to task outcomes, including those of summary-based methods. Once a question incurs a loss, concentrating demand on that question rules out universal dominance. The rank-based analysis in Appendix C.2 explains a mechanism for such losses when evidence rankings change. DREAM responds by using recent questions to allocate retrieval entrances to the user's observed needs, while preserving the original evidence.