acceptodds
Under review as a conference paper at ICLR 2027

R²-Mem: Reflective Experience for Memory Search

Abstract

Traditional memory systems often organize or compress past information before a query arrives, which may leave out details needed to answer it. Deep memory search addresses this problem by searching full historical records on demand through planning, retrieval, and reflection. However, final outcomes alone do not show which search behaviors to keep or correct. Successful searches may include unnecessary steps, while failed searches may still contain useful ones. We propose R^2-Mem, a reflective experience framework for iterative memory search. In the offline stage, a Rubric-guided Evaluator assesses planning and reflection steps using search-specific, multi-dimensional rubrics, providing scores, reasons, and practical suggestions. A self-Reflection Learner then turns selected high- and low-scoring steps and their feedback into reusable fine-grained reflective experience. During inference, the agent retrieves relevant experience to guide planning and reflection without updating model parameters. Experiments on LoCoMo, HotpotQA, and NarrativeQA show consistent gains in answer quality in agent memory tasks. With Qwen2.5-3B, R^2-Mem achieves a 22.6% relative F1 gain, while reducing online token usage by 12.9% and search iterations by 20.2% Experiments with Self-Evo further show that the backbone can achieve experience-based self-evolution through self-evaluation, without a stronger external evaluator.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Agent memory, Experience learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jN6HOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes R²-Mem, a training-free framework that improves iterative "deep memory search" agents by learning from past search trajectories. Offline, a Rubric-guided Evaluator scores each Planning and Reflection step on module-specific 0–3 rubric dimensions and emits diagnostic reasons/advice; a Self-Reflection Learner distills high-scoring and low-scoring steps into abstract IF-THEN "experiences" stored in separate planning and reflection banks indexed by an abstracted situation embedding. At inference, top-k experiences are retrieved by similarity on the current query/memory state and injected into the corresponding stage, with no parameter updates.

Experiments on LoCoMo, HotpotQA, and NarrativeQA with Qwen2.5-3B/7B/14B and Llama3.1-8B show consistent F1/BLEU1 gains over GAM and over structured-memory baselines (A-Mem, Mem0, MemoryOS, LightMem) and Memory-R1, with reduced online tokens and iterations. Ablations separate the Evaluator, Learner, high- vs low-quality experience, and planning vs reflection banks.

Critical strengths
  1. Careful component-level validation of the central claim. The ablations in Table 4 directly test the paper's thesis that step-level, rubric-guided diagnosis matters.
  2. Efficiency and quality improve simultaneously, with the offline cost quantified rather than hidden. Table 3 shows F1 gains together with fewer online tokens and iterations across all three backbone sizes.
Critical weaknesses
  1. "Experience" is benchmark-specific and answer-supervised, so the reusability/generality claim is not established. Experience banks are built per-benchmark from in-distribution questions. There is no cross-dataset or cross-domain transfer experiment (e.g., LoCoMo-built bank applied to HotpotQA), so the paper cannot distinguish "reusable fine-grained reflective experience" (Abstract, Sec. 4.3) from within-distribution adaptation to a benchmark's question style — a concern amplified on LoCoMo, where one conversation's distilled heuristics (e.g., the temporal-query patterns in Appendix E.2) transparently match the other nine conversations' dominant category mix (Table 5). Relatedly, no published experience-learning method is compared directly; the only point of comparison is the authors' own "w/o Evaluator" re-implementation described as ReasoningBank-style.
Questions for authors
  1. (Clarification) For Table 1, which baselines were run once on which split (B.2), and what were the "preliminary runs" showing "only minor differences"? Rerunning at least Memory-R1 and A-Mem on all three splits would resolve the fairness concern.
  2. (Suggestion) Does experience transfer across benchmarks or question distributions? E.g., apply the LoCoMo bank to HotpotQA (or Conv-26's bank to a held-out LoCoMo category absent from the source conversation) to substantiate "reusable" beyond in-distribution.
  3. (Clarification) The Evaluator consumes REFERENCE_ANSWER (D.1). How much does performance degrade if rubric scoring is done without gold answers, which would be the realistic deployment condition implied by Sec. F.2?
  4. (Clarification) Appendix C.1 compares token counts across different models (GPT-4o offline vs Qwen online). What is the comparison in dollar cost or FLOPs, and does the amortization claim for 14B (0.53M offline vs 1.12M saved) survive GPT-4o pricing?
  5. (Suggestion) Why not compare directly against ReasoningBank/ExpeL adapted to GAM, rather than only your own "w/o Evaluator" approximation (Sec. 5.4)?
1 comment

Thank you very much for your response and detailed clarification. Much of my concern has been addressed. I lean towards acceptance and maintain my initial assessment of weak acceptance. Wish you good luck with your submission.

Reviewer_xkCcOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

R²-Mem addresses the coarse granularity of trajectory-level experience learning in iterative deep memory search. A successful trajectory may contain redundant or incorrect steps, while a failed trajectory may still include useful planning, so final outcomes alone cannot determine which intermediate behaviors should be reused. R²-Mem applies module-specific rubrics to planning and reflection steps, producing scores, diagnostic reasons, and actionable advice. A Self-Reflection Learner abstracts high-scoring steps into good practices and low-scoring steps into corrective experiences, storing them in separate planning and reflection experience banks. During online search, relevant experiences are retrieved based on the current query or the query together with temporary memory and injected into a GAM-style search loop without updating the backbone. Experiments on LoCoMo, HotpotQA, and NarrativeQA generally improve answer quality while reducing online tokens and search iterations, and Self-Evo shows that backbone self-evaluation retains much of the gain.

Critical strengths
  1. R²-Mem evaluates planning and reflection steps separately instead of assigning every step the trajectory-level outcome. It distills high-scoring steps into reusable practices and low-scoring steps into corrective experiences, thereby retaining useful behavior from failed trajectories and revising redundant behavior in successful ones.
  2. Across LoCoMo, HotpotQA, NarrativeQA, and multiple Qwen2.5 and Llama backbones, R²-Mem generally outperforms the direct GAM baseline while reducing online token usage and search iterations. Ablations over high- versus low-scoring experience, planning versus reflection, and the Evaluator versus Learner provide useful internal evidence.
  3. The method augments search through natural-language experience banks without updating the task-performing backbone. Self-Evo remains stronger than GAM when the backbone evaluates its own steps, suggesting that the gains are not wholly dependent on GPT-4o supervision and may transfer across evaluator choices.
Critical weaknesses
  1. The backbone study uses Qwen2.5-3B/7B/14B and Llama3.1-8B, which are relatively dated. No results are reported with newer Qwen3- or Qwen3.5-class backbones, so the reported gains may not reflect current model generations.
  2. The appendix contains a JSON Output Format block that overflows its page/section boundary; the appendix layout and content should be carefully checked. The reference list is relatively short and does not sufficiently cover relevant recent work.
Questions for authors
  1. Can the authors provide benchmarks on newer backbones, such as Qwen3 or Qwen3.5, to show that the gains over GAM are not specific to Qwen2.5/Llama3.1?
  2. Can the authors evaluate R²-Mem on Code-Agent benchmarks such as SWE-Bench Verified and Terminal-Bench 2.0, reporting task success and search efficiency to verify whether the experience-based gains transfer to code retrieval and tool-use settings?
1 comment

Thank you for the additional experiments and clarifications. They address my main concerns, although I still encourage the authors to add the Code-Agent evaluation in a future revision.

Reviewer_jUH2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to improve deep memory search by looking at the model’s past searches and learning from both its mistakes and what it did well. It then adds relevant lessons to the prompt to help the model avoid repeating those mistakes, make better searches, and decide when it has enough information to answer.

Critical strengths
  1. The paper tries to solve an interesting problem.
  2. The method is inherently simple and hence can be deployed without much effort.
Critical weaknesses
  1. My main concern is that the paper does not compare against GPT-4o directly solving the same tasks with comparable access to the retrieval tools. The results show that the proposed method improves the smaller models, but it is still unclear whether this gives a better tradeoff between answer quality and cost than simply using a stronger model. Token counts alone do not settle this, since tokens from different models have different costs.

  2. This is a minor presentation issue: Section 5.3 identifies the gains as relative improvements, but this should also be explicit in the relevant tables.

  3. The paper gives the LoCoMo question counts, but these questions come from only ten conversation histories. Even though there are many questions, questions from the same conversation share the same context. It is therefore important to know whether the gains are consistent across conversations or mainly come from a few of them. Since the authors also use different conversations to build the experience bank, reporting standard deviation across the three splits would help show how much the results depend on that choice. For HotpotQA and NarrativeQA, I could not find the exact numbers of questions used for experience construction and evaluation in the submitted version.

Questions for authors
  1. Could the authors include a GPT-4o baseline with comparable retrieval access and report answer quality, cost, and latency? How does the comparison change when the cost of building the experience bank is included?

  2. Could the authors report the standard deviation across runs across datasets? Could they also give the exact number of questions used for experience construction and evaluation for each HotpotQA context-length setting and for NarrativeQA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.