acceptodds
Under review as a conference paper at ICLR 2027

LocalMem: Learning to Archive and Retrieve Long-Term Memory for LLM Agents

Abstract

Long-term interactive agents need external memory to retain information across interactions and recover information required by later queries. Existing systems commonly use fixed memory-management rules and general-purpose large language models (LLMs), e.g., GPT-4o, for memory operations. However, fixed rules often store or retrieve misaligned memory evidence, while reliance on cloud-hosted LLMs raises operating costs and privacy risks. We present LocalMem, a plug-and-play memory-management system with separate local language models for memory construction and retrieval. For memory construction, a 4B reinforced Memory Archiver is optimized with the proposed unified memory reward to learn what information to preserve. For memory retrieval, a 4B SFT-trained Memory Enquirer uses progressive retrieval to acquire additional evidence only when needed, improving the accuracy-efficiency trade-off. The proposed LocalMem achieves the state-of-the-art (SOTA) performance, i.e., 69.3 Overall F1 on LoCoMo10 and 81.8% LLM-as-a-Judge accuracy on LongMemEval-S. Controlled ablations show that the trained 4B Archiver outperforms a prompted 122B Archiver by 4.1 F1, while progressive retrieval improves F1 by 1.5 over fixed top- retrieval.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Long-Term Memory, Adaptive Retrieval, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ixFaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents LocalMem, a memory-management framework with separate 4B models for memory construction and retrieval. The Memory Archiver is trained with SFT and GRPO using downstream QA rewards. The framework runs a four-stage retrieval pipeline guided by evidence-sufficiency judgments.

Critical strengths
  • The problem is relevant: improving long-term memory for LLM agents while reducing memory-management costs.

  • The paper is generally clearly written.

Critical weaknesses
  • The novelty appears limited, as the paper mainly combines existing techniques. SimpleMem already uses window-based extraction and semantic, lexical, and structured indexing, while Mem-α learns memory construction from downstream QA feedback with frozen retrieval and generation components. Adaptive and iterative retrieval have been explored in existing studies like Self-RAG and IRCoT. The paper does not sufficiently establish the contribution of its particular combination beyond these existing ideas. I think using local 4B models is primarily an engineering choice and does not establish methodological novelty.

  • The gain of reinforcement learning appears modest. Table 4 shows that SFT provides most of the improvement, increasing F1 from 55.0 to 60.5, while GRPO further increases it to 61.1. These results do not clearly show whether the additional RL stage and the proposed reward formulation are necessary. Additionally, the evaluation covers only two conversational memory benchmarks. It is therefore unclear whether the learned memory policies benefit agents performing interactive tasks, where memory must support action selection and errors can affect subsequent observations. Results in other settings, such as ALFWorld, would help assess the framework’s broader applicability to LLM agents.

Questions for authors
  • How does independent window processing handle cross-window coreferences and changes in facts over time, particularly when new information contradicts or supersedes previously recorded facts?
5 comments
AuthorsOpenReview ↗

To Reviewer ixFa/#3 (1/3)

Concern 1 (Weakness 1): The novelty appears limited, as the paper mainly combines existing techniques. SimpleMem already uses window-based extraction and semantic, lexical, and structured indexing, while Mem-α learns memory construction from downstream QA feedback with frozen retrieval and generation components.

Response 1: We thank the reviewer for questioning the novelty of LocalMem relative to SimpleMem and Mem-α. We agree that window-based extraction, multi-view indexing, and downstream QA supervision are not individually new. Our contribution lies in addressing the mismatch between independent window-level memory generation and conversation-level memory utility through a unified memory reward.

SimpleMem already employs window-based extraction and multi-view indexing, while Mem-α uses downstream QA feedback to optimize memory construction. LocalMem adopts Streaming Window-to-Entry Generation, where each interaction window independently generates structured memory entries without accessing or modifying previously stored memories.

This design introduces a granularity mismatch: memory utility is evaluated jointly at the conversation level, whereas memory generation occurs independently at the window level. We address this mismatch through a unified memory reward that evaluates the assembled memory according to downstream QA performance. We further introduce Window-count Normalization to balance conversations with different numbers of valid windows and Format Reward to constrain the structure of generated entries.

Window-count NormalizationFormat RewardOverall₄
✓✓61.03 ± 0.26
×✓59.32 ± 0.16
✓×60.73 ± 0.29

Removing Window-count Normalization and Format Reward decreases Overall₄ F1 by 1.71 and 0.30 points, respectively.

Therefore, our contribution is not the first use of QA feedback for memory learning, but an optimization mechanism tailored to independent window-level memory generation that aligns window-level learning with conversation-level memory utility.

Concern 2 (Weakness 1): Adaptive and iterative retrieval have been explored in existing studies like Self-RAG and IRCoT. The paper does not sufficiently establish the contribution of its particular combination beyond these existing ideas.

Response 2:

We thank the reviewer for pointing out the need to clarify the distinction between LocalMem and adaptive retrieval methods such as Self-RAG and IRCoT. We agree that adaptive and iterative retrieval have been extensively studied. However, LocalMem uses evidence sufficiency relative to the original information requirements to guide subsequent memory retrieval.

Self-RAG uses reflection signals to assess retrieval requirements and document relevance, while IRCoT generates subsequent queries based on intermediate reasoning steps. In contrast, LocalMem employs a frozen Evidence-sufficiency Judge to determine whether retrieved memories sufficiently cover the original information requirements. When the evidence is insufficient, the Memory Enquirer explicitly identifies the missing information and generates targeted retrieval queries.

We further isolate the contribution of Gap-aware Planning through a controlled experiment. With the expanded Top-K settings held constant, enabling Gap-aware Planning improves Overall₄ F1 from 60.67 to 61.39 (+0.72 points).

Retrieval StrategyExpand Top-KGap PlanOverall₄Δ Overall₄
Tier 1–3××59.22 ± 0.04—
Tier 1–3 + Expanded Tier-3 Top-K✓×60.67 ± 0.09+1.45
Tier 1–4✓✓61.39 ± 0.15+2.17

This controlled comparison demonstrates that targeted query refinement based on evidence gaps provides additional gains beyond merely increasing the retrieval candidate budget.

Concern 3 (Weakness 1): I think using local 4B models is primarily an engineering choice and does not establish methodological novelty.

Response 3: We thank the reviewer for pointing out that using compact 4B models is primarily an engineering choice. We agree that model size itself is not the source of methodological novelty. Our contributions lie in the memory learning objectives and retrieval mechanisms, while compact models provide an efficient implementation of these mechanisms.

For the Memory Archiver, the trained 4B model improves over its prompted 122B counterpart by 4.1 F1 points. For the Memory Enquirer, the trained 4B model achieves 61.3 ± 0.2 F1, approaching the 61.2 ± 0.2 achieved by the 122B Progressive Enquirer, while outperforming the untrained 4B Enquirer (59.7 F1).

These results demonstrate that the proposed memory mechanisms can be learned effectively with compact models, rather than requiring substantially larger models for memory construction and retrieval.

We therefore position compact models as an efficient implementation of the proposed memory mechanisms, rather than as the basis of methodological novelty.

AuthorsOpenReview ↗

To Reviewer ixFa/#3 (2/3)

Concern 4 (Weakness 2): The gain of reinforcement learning appears modest. Table 4 shows that SFT provides most of the improvement, increasing F1 from 55.0 to 60.5, while GRPO further increases it to 61.1. These results do not clearly show whether the additional RL stage and the proposed reward formulation are necessary.

Response 4: We thank the reviewer for questioning the necessity of the additional reinforcement learning stage. We agree that SFT accounts for most of the overall improvement, while GRPO provides additional benefits on challenging multi-hop questions and memory compactness.

Specifically, GRPO improves Overall₄ F1 from 60.54 to 61.03 (+0.49 points), including a gain of 1.34 points on MH questions. Meanwhile, Memory Retention (the ratio of generated memory tokens to the total number of original conversation tokens) decreases from 72.70% to 70.08%.

Memory ArchiverMHTempODSHOverall₄
SFT46.71 ± 0.6262.00 ± 0.2729.09 ± 0.9968.20 ± 0.3160.54 ± 0.16
+ GRPO48.05 ± 0.3462.64 ± 0.4829.79 ± 1.9468.33 ± 0.0461.03 ± 0.26

The reward ablations further show that removing Window-count Normalization and Format Reward decreases Overall₄ F1 by 1.71 and 0.30 points, respectively.

Window-count NormalizationFormat RewardMHTempODSHOverall₄
✓✓48.05 ± 0.3462.64 ± 0.4829.79 ± 1.9468.33 ± 0.0461.03 ± 0.26
×✓45.03 ± 0.2260.45 ± 0.8530.48 ± 1.5766.98 ± 0.5059.32 ± 0.16
✓×46.86 ± 0.7261.88 ± 1.1331.46 ± 1.8368.27 ± 0.1260.73 ± 0.29

Together, these results indicate that although GRPO contributes a modest additional gain in aggregate F1, it improves more challenging multi-hop performance while encouraging more compact and structurally consistent memory representations.

Concern 5 (Weakness 2): Additionally, the evaluation covers only two conversational memory benchmarks. It is therefore unclear whether the learned memory policies benefit agents performing interactive tasks, where memory must support action selection and errors can affect subsequent observations. Results in other settings, such as ALFWorld, would help assess the framework’s broader applicability to LLM agents.

Response 5: We thank the reviewer for pointing out that the original evaluation focuses mainly on conversational memory and does not sufficiently examine its support for Agent tasks. We therefore extend our evaluation to Mem2ActBench [1], which directly tests whether an Agent can recover complete and executable tool-call parameters from long-term memory.

[1] Shen Y, Li K, Zhou W, et al. Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents.

Using the shared Qwen2.5-72B Answer Model, LocalMem-4B achieves 39.13 F1 and 70.24 BLEU-1, outperforming the compared memory baselines.

MethodMemory ModelAnswer ModelF1 ↑BLEU-1 ↑
LocalMem (Ours)LocalMem-4BQwen2.5-72B39.1370.24
A-MemQwen2.5-72BQwen2.5-72B35.9367.93
LTMemoryQwen2.5-72BQwen2.5-72B35.3267.93
MemTreeQwen2.5-72BQwen2.5-72B33.2168.17
Mem0Qwen2.5-72BQwen2.5-72B28.9566.47
LangmemQwen2.5-72BQwen2.5-72B24.0163.06
SCM—Qwen2.5-72B22.7362.04
Generative AgentsQwen2.5-72BQwen2.5-72B22.3862.58

We select Mem2ActBench because it directly evaluates the recovery of parameters required for downstream tool execution from long-term memory. In contrast, ALFWorld primarily evaluates closed-loop embodied interaction and environment-level task execution, involving broader reasoning, planning, and action-generation capabilities beyond the direct scope of LocalMem as a memory-management module.

This additional evaluation demonstrates that LocalMem can support memory-grounded tool use beyond conventional conversational QA, while we acknowledge that it does not constitute a full evaluation of closed-loop interactive environments.

AuthorsOpenReview ↗

To Reviewer ixFa/#3 (3/3)

Concern 6 (Question 1): How does independent window processing handle cross-window coreferences and changes in facts over time, particularly when new information contradicts or supersedes previously recorded facts?

Response 6: We thank the reviewer for raising these important questions about cross-window coreference and temporal changes in stored facts. LocalMem addresses cross-window references through overlapping windows and preserves temporal changes through an append-only memory archive.

For cross-window coreference, adjacent windows share (k=2) dialogue turns. Within each window, the Memory Archiver resolves pronouns into explicit entity names whenever their antecedents can be identified from the available context. This overlapping context helps preserve references near window boundaries while maintaining independent and parallel memory generation. However, references may remain unresolved when their antecedents fall outside the available overlapping context.

For temporal changes, LocalMem stores new information as separate memory entries with associated timestamps rather than overwriting previous records. Thus, both earlier and updated or contradictory information are preserved. LocalMem does not maintain a globally reconciled, unique fact state. Instead, the downstream Answer Model uses retrieved entries and their timestamps to interpret temporal changes and resolve potential contradictions according to the query.

This design preserves historical information without destructive updates, while delegating temporal interpretation and contradiction resolution to downstream reasoning.

Reviewer_4n6DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

This paper presents LocalMem, a long-term conversational memory system that learns memory construction and retrieval using two separate 4B language models. The Memory Archiver independently converts overlapping interaction windows into structured atomic memory entries containing a lossless restatement, keywords, and optional temporal/entity/location metadata. It is first trained by supervised distillation and then optimized with GRPO using a unified conversation-level downstream QA reward shared across the window-level generations. The Memory Enquirer is SFT-trained to support progressive retrieval. Retrieval begins with small dense-retrieval tiers, then invokes the Enquirer for multi-view planning and gap-aware refinement only when a frozen evidence-sufficiency judge finds the accumulated evidence inadequate. A separate frozen Answer Model produces the final answer.

The system is trained on synthetic conversations from the Mem-α training set and evaluated on LoCoMo10 and LongMemEval-S. With Qwen3.5-122B as the judge/answer model, the paper reports 69.3 Overall5 F1 on LoCoMo10 and 81.8% LLM-as-a-Judge accuracy on LongMemEval-S. Controlled experiments compare prompted and trained archivers, fixed and progressive retrieval, retrieval depths, different sufficiency judges, and different answer backbones. The paper also reports construction tokens/calls, retrieval latency, routing behavior, and a preliminary experiment consolidating the two 4B policies.

Critical strengths
  1. The paper offers a coherent and practically motivated learned memory pipeline. The independent window-to-entry Archiver avoids sequential edits to a growing memory state, while progressive retrieval explicitly tests evidence sufficiency and reserves more expensive planning for harder questions. This division is technically clean and addresses both write-side and read-side costs.

  2. The empirical validation is strong and substantially more controlled than a simple cross-paper SOTA table. The paper includes matched-answer-backbone comparisons, prompted-versus-trained Archiver comparisons, fixed-versus-progressive retrieval, retrieval-depth ablations, alternative sufficiency judges, cross-backbone transfer, construction-cost measurements, and detailed routing statistics. The result that a trained 4B Archiver and Enquirer can match or outperform much larger prompted models is consequential for the feasibility of specialized memory models.

Critical weaknesses
  1. The main results do not evaluate a fully local system. The trained Archiver and Enquirer are 4B models, but the headline pipeline still uses a 122B model for sufficiency judging and final answering. An end-to-end local evaluation with accuracy and hardware costs is needed to support the local/mobile/privacy claims.

  2. The gains from the proposed training are small and lack statistical support. GRPO adds only 0.5 F1 over SFT, and the trained Enquirer adds only 0.1 F1 over the 122B progressive Enquirer. The paper should report per-run uncertainty and matched comparisons with stronger adaptive or learned-memory baselines.

  3. The evaluation is too narrow to support broad plug-and-play claims. The main experiments cover only two text-based conversational-memory benchmarks, with LoCoMo10 containing ten conversations. Evaluation on additional datasets, agent tasks, and smaller answer models is needed to establish generalization beyond the current setup.

Questions for authors
  1. How well does a fully local end-to-end system perform without the 122B judge and answer model? Please include latency and memory requirements.

  2. Are the 0.5-F1 GRPO gain and 0.1-F1 Enquirer gain statistically reliable across runs?

  3. Can you evaluate against stronger adaptive or learned-memory baselines under the same answer model and on additional benchmarks?

5 comments
AuthorsOpenReview ↗

To Reviewer 4n6D/#2 (1/2)

Concern 1 (Weakness 1 & Question 1): The main results do not evaluate a fully local system. The trained Archiver and Enquirer are 4B models, but the headline pipeline still uses a 122B model for sufficiency judging and final answering. An end-to-end local evaluation with accuracy and hardware costs is needed to support the local/mobile/privacy claims. How well does a fully local end-to-end system perform without the 122B judge and answer model? Please include latency and memory requirements.

Response 1: We thank the reviewer for raising this important concern. We agree that our main evaluation does not represent a fully local end-to-end system, as it employs a 122B model for evidence-sufficiency judging and answer generation. To examine the feasibility of using compact models throughout the pipeline, we additionally evaluate a 4B/4B configuration, in which memory construction, retrieval, evidence-sufficiency judging, and answer generation are all performed using 4B models.

Memory ModelJudge/Answer ModelMHTempODSHOverall₄
4B4B43.05 ± 0.1459.55 ± 0.2216.04 ± 0.7262.78 ± 0.0455.58 ± 0.05
4B122B48.45 ± 0.4163.29 ± 0.3930.77 ± 1.4468.29 ± 0.3561.28 ± 0.21

The 4B/4B configuration achieves an Overall₄ F1 of 55.58, demonstrating that LocalMem retains useful long-term memory capabilities when compact models are used throughout the pipeline.

Importantly, the central goal of LocalMem is to provide a local, plug-and-play memory-management module rather than a standalone end-to-end Agent. It separates long-term memory management from downstream reasoning and task execution: the Memory Archiver locally converts interaction histories into structured memory entries, while the Memory Enquirer retrieves relevant information and identifies evidence gaps. The host LLM or Agent then uses the retrieved memories for downstream reasoning and task execution.

We therefore distinguish the local execution of memory management from the deployment of the complete host system. While the 4B/4B experiment demonstrates the feasibility of an all-compact-model pipeline, our primary contribution is a locally executable memory module that can be integrated with different downstream models.

Concern 2 (Weakness 2 & Question 2): The gains from the proposed training are small and lack statistical support. GRPO adds only 0.5 F1 over SFT, and the trained Enquirer adds only 0.1 F1 over the 122B progressive Enquirer. The paper should report per-run uncertainty. Are the 0.5-F1 GRPO gain and 0.1-F1 Enquirer gain statistically reliable across runs?

Response 2: We thank the reviewer for raising concerns about the training gains and their statistical reliability. We provide repeated-evaluation standard deviations and an additional independent training run, while carefully interpreting the small performance difference between the 4B Enquirer and the 122B Progressive Enquirer.

Specifically, GRPO improves Overall₄ F1 from 60.54 to 61.03 (+0.49 points). The gain is more pronounced on Multi-Hop (MH) questions (+1.34 points), compared with +0.13 points on Single-Hop (SH) questions. Notably, MH questions account for only 18.3% of the evaluation set (282/1,540), limiting their contribution to the overall score. Consequently, the substantial improvement on MH questions is diluted in the aggregate Overall₄ F1, resulting in a relatively modest overall gain.

Memory ArchiverMHTempODSHOverall₄
SFT46.71 ± 0.6262.00 ± 0.2729.09 ± 0.9968.20 ± 0.3160.54 ± 0.16
+ GRPO, seed=4248.05 ± 0.3462.64 ± 0.4829.79 ± 1.9468.33 ± 0.0461.03 ± 0.26
Δ+1.34+0.64+0.70+0.13+0.49

These means and standard deviations are computed over three repeated evaluations of the same trained models, rather than independent training runs. To provide further evidence of training consistency, we independently retrained the SFT+GRPO Archiver using seed=1, achieving an Overall₄ F1 of 61.06.

Training RunMHTempODSHOverall₄
SFT46.71 ± 0.6262.00 ± 0.2729.09 ± 0.9968.20 ± 0.3160.54 ± 0.16
+ GRPO, seed=148.0363.5130.0968.0361.06

For the Memory Enquirer, the trained 4B model achieves 61.3 ± 0.2 F1, comparable to 61.2 ± 0.2 F1 achieved by the 122B Progressive Enquirer. Given the small difference, we do not claim that the 4B model significantly outperforms its 122B counterpart. Instead, we emphasize that the compact model approaches the retrieval performance of a substantially larger model.

Memory EnquirerOverall₄ F1
Untrained 4B Enquirer59.7
Trained 4B Enquirer61.3 ± 0.2
122B Progressive Enquirer61.2 ± 0.2

Additionally, the trained 4B Enquirer improves over the untrained 4B Enquirer by 1.6 points.

AuthorsOpenReview ↗

To Reviewer 4n6D/#2 (2/2)

Concern 3 (Weakness 3 & Question 3): The evaluation is too narrow to support broad plug-and-play claims. The main experiments cover only two text-based conversational-memory benchmarks, with LoCoMo10 containing ten conversations. Evaluation on additional datasets and agent tasks is needed to establish generalization beyond the current setup.

Response 3: We thank the reviewer for pointing out the limited scope of our original evaluation. We agree that additional experiments are needed to examine the applicability of LocalMem beyond text-based conversational memory. We therefore extend our evaluation to multimodal conversational memory and memory-grounded Agent tasks, while retaining the original text-trained 4B Memory Archiver and Memory Enquirer.

(1) Multimodal Conversational Memory. We extend our experiments to Mem-Gallery [1], a multimodal conversational-memory benchmark. We introduce a media_refs field to associate memory entries with image assets and employ separate text and visual retrieval channels. Caption embeddings use a 1024-dimensional text embedding space, while SigLIP2 image embeddings form an independent 768-dimensional visual embedding space.

[1] Bei Y, Wei T, Ning X, et al. Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents.

Under the shared Qwen2.5-VL-7B Answer Model, LocalMem-4B achieves:

MethodMemory ModelAnswer ModelF1 ↑BLEU-1 ↑
LocalMem (Ours)LocalMem-4BQwen2.5-VL-7B62.6456.95
A-MemQwen2.5-VL-7BQwen2.5-VL-7B62.2856.29
MemoryOSQwen2.5-VL-7BQwen2.5-VL-7B61.0955.34
MemGPTQwen2.5-VL-7BQwen2.5-VL-7B52.8247.92

LocalMem achieves the best results among the compared methods on both metrics. This experiment demonstrates that LocalMem can be adapted to multimodal memory through image-linked entries and an independent visual retrieval channel, without modifying its core memory architecture or requiring additional training of the memory models.

(2) Memory-grounded Agent Tasks. We further evaluate memory-grounded tool use on Mem2ActBench [2], which examines whether an Agent can recover complete and executable tool calls from long-term memory, including parameters stored in previous interactions.

[2] Shen Y, Li K, Zhou W, et al. Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents.

Using the shared Qwen2.5-72B Answer Model, LocalMem-4B achieves 39.13 F1 and 70.24 BLEU-1, outperforming all compared memory baselines on both metrics.

MethodMemory ModelAnswer ModelF1 ↑BLEU-1 ↑
LocalMem (Ours)LocalMem-4BQwen2.5-72B39.1370.24
A-MemQwen2.5-72BQwen2.5-72B35.9367.93
LTMemoryQwen2.5-72BQwen2.5-72B35.3267.93
MemTreeQwen2.5-72BQwen2.5-72B33.2168.17
Mem0Qwen2.5-72BQwen2.5-72B28.9566.47
LangmemQwen2.5-72BQwen2.5-72B24.0163.06
SCM—Qwen2.5-72B22.7362.04
Generative AgentsQwen2.5-72BQwen2.5-72B22.3862.58

We select Mem2ActBench because it directly evaluates whether an Agent can recover parameters required for downstream tool execution from long-term memory. Unlike ALFWorld, which primarily evaluates closed-loop embodied interaction and environment-level task execution, Mem2ActBench focuses on the memory-retrieval capability that is directly within the scope of LocalMem.

Together, these experiments extend our evaluation beyond conventional text-based conversational memory to multimodal memory and memory-grounded tool use, while keeping the memory models unchanged.

Concern 4 (Weakness 2 & Question 3): The paper should report matched comparisons with stronger adaptive or learned-memory baselines. Can you evaluate against stronger adaptive or learned-memory baselines under the same answer model and on additional benchmarks?

Response 4: We thank the reviewer for suggesting matched comparisons with stronger memory baselines. In our main experiments, we evaluate different memory methods using the same Qwen3.5-122B Answer Model.

MethodAnswer ModelOverall₄ F1
LocalMem (Ours)Qwen3.5-122B61.3
Omni-SimpleMemQwen3.5-122B51.6
Mem0Qwen3.5-122B50.3

LocalMem achieves an Overall₄ F1 of 61.3, outperforming both Omni-SimpleMem and latest Mem0 under the same Answer Model.

We further conduct matched-Answer-Model comparisons on Mem-Gallery and Mem2ActBench, as reported in Response 3. These additional experiments evaluate LocalMem against stronger memory baselines in multimodal conversational memory and memory-grounded Agent tasks, extending the comparison beyond the original text-based conversational-memory setting.

We express our sincere gratitude to the reviewer for dedicating time to review our paper. We have provided comprehensive responses to all the concerns. As the discussion deadline looms within 2 days, we would like to inquire if our responses have adequately addressed your questions. We are more than willing to address any concerns and ensure a comprehensive resolution. Thank you for your time and consideration.

Reviewer_66gMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LocalMem, a modular memory-management system with separately trained 4B models for memory construction and retrieval. The Memory Archiver independently converts interaction windows into structured entries and is trained with supervised fine-tuning followed by GRPO using conversation-level question-answering rewards. The Memory Enquirer is trained through supervised distillation to generate retrieval plans and gap refinements. A four-tier retrieval procedure combines dense retrieval, multi-view retrieval, and refinement, with a frozen evidence-sufficiency judge controlling early stopping. It reports results on LoCoMo10 and LongMemEval-S, including matched-answer-model comparisons, component ablations, retrieval-depth analyses, and experiments with alternative sufficiency judges. Its main empirical finding is that specialized 4B memory models can outperform larger prompted memory components while reducing memory-management latency.

Critical strengths
  1. A practical and modular design. Independent window-to-entry generation avoids sequential memory editing during construction and permits parallel processing. Separating memory management from answer generation also allows the trained memory components to be reused with different downstream models.

  2. Meaningful comparisons under a shared Answer Model. In Table 2, LocalMem achieves 61.3 Overall₄ F1, compared with 51.6 for Omni-SimpleMem and 50.3 for Mem0, using the same Qwen3.5-122B Answer Model.

  3. Useful component ablations. Table 4 shows that the trained 4B Archiver outperforms both its untrained counterpart and a prompted 122B Archiver with the retrieval and answering components held fixed. Table 5 separately evaluates retrieval strategy and Enquirer training, showing that the trained 4B Enquirer closely matches the 122B progressive Enquirer while reducing reported retrieval latency from 6.7 to 4.2 seconds.

Critical weaknesses
  1. The specific contribution of the unified memory reward is relatively limited and weakly isolated. Empirically, most of the Archiver improvement in Table 4 comes from SFT: Overall₄ increases from 55.0 to 60.5, while GRPO further increases it to 61.1. The paper reports averages over three runs but provides no dispersion estimates. The additional benefit of GRPO, particularly the proposed normalization and format-reward treatment, is therefore less firmly established than the benefit of specialist training overall.

  2. The retrieval ablations do not fully isolate evidence-based stopping and gap-aware refinement. Table 5 describes its standard comparator as a SimpleMem-style fixed-top-kk strategy. However, SimpleMem explicitly includes adaptive or intent-aware retrieval planning, so the exact implementation, retrieval budget, and enabled features of this comparator need clarification. In addition, progressing from Tier 3 to Tier 4 changes both the retrieval queries and the candidate budgets. The improvement in Table 6 therefore does not separately establish the value of gap-aware planning. A focused control that expands the same retrieval queries to the Tier-4 budget without generating a gap plan would help identify this contribution. The existing results support the complete progressive pipeline more directly than its individual mechanisms.

Questions for authors
  1. What variability is observed across the three runs for the SFT and SFT+GRPO Archivers? Are these independent training runs? Is there evidence isolating the benefit of the 1/Ni1/N_i normalization or the window-specific format penalty?

  2. What exact SimpleMem implementation and configuration constitute the standard retrieval baseline in Table 5? How does Tier 4 compare with increasing the candidate budget while retaining the Tier-3 queries?

  3. Please identify the cleaned evaluation split/version, any exclusions, the adversarial-question scoring procedure, and the validation procedure used to select checkpoints and retrieval settings.

  4. How can Tier 4 recover a previously searched but still unsupported information need under the prompt in Appendix A.8.4? Please provide an example including the original requirements, attempted requirements, returned gap plan, and newly retrieved evidence.

4 comments
AuthorsOpenReview ↗

To Reviewer 66gM/#1 (1/3)

Concern 1 (Weakness 1): The specific contribution of the unified memory reward is relatively limited and weakly isolated. Empirically, most of the Archiver improvement in Table 4 comes from SFT: Overall₄ increases from 55.0 to 60.5, while GRPO further increases it to 61.1.

Response 1: We thank the reviewer for pointing out the relatively modest additional gain of GRPO over SFT. We agree that SFT accounts for most of the overall improvement, while GRPO provides complementary benefits, particularly on challenging multi-hop questions, and reduces memory-token usage. Specifically, GRPO improves Overall₄ F1 from 60.54 to 61.03 (+0.49 points). The gain is more pronounced on Multi-Hop (MH) questions (+1.34 points), compared with only +0.13 points on Single-Hop (SH) questions. Notably, MH questions account for only 18.3% of the evaluation set (282/1,540), so their improvement contributes relatively little to the aggregate Overall₄ score.

Memory ArchiverMHTempODSHOverall₄
SFT46.71 ± 0.6262.00 ± 0.2729.09 ± 0.9968.20 ± 0.3160.54 ± 0.16
+ GRPO48.05 ± 0.3462.64 ± 0.4829.79 ± 1.9468.33 ± 0.0461.03 ± 0.26
Δ+1.34+0.64+0.70+0.13+0.49

We also examine memory compactness by defining Memory Retention as the ratio of generated memory tokens to the total number of original conversation tokens. GRPO reduces this ratio from 72.70% to 70.08% (−2.62 percentage points), indicating that the Archiver preserves task-relevant information using fewer memory tokens while maintaining or improving downstream QA performance.

Memory ArchiverMemory Retention ↓
SFT72.70%
+ GRPO70.08%
Δ−2.62 pp

Overall, these results suggest that GRPO provides complementary benefits beyond SFT: improving performance on challenging multi-hop questions while encouraging more compact memory representations.

Concern 2 (Weakness 1 & Question 1): The paper reports averages over three runs but provides no dispersion estimates. What variability is observed across the three runs for the SFT and SFT+GRPO Archivers? Are these independent training runs?

Response 2: We appreciate this comment. We clarify that our reported standard deviations reflect repeated evaluations of the same trained checkpoints, rather than independent training runs. Specifically, the means and standard deviations reported in Response 1 are computed over three independent evaluations of the same SFT and SFT+GRPO models. To provide an additional check on training reproducibility, we conducted an independent SFT+GRPO training run using seed=1. It achieves an Overall₄ F1 of 61.06, close to the original evaluation mean of 61.03.

Training ConfigurationMHTempODSHOverall₄
SFT+GRPO, original checkpoint (seed=42)48.0562.6429.7968.3361.03
SFT+GRPO, independent training (seed=1)48.0363.5130.0968.0361.06

This additional run provides preliminary evidence that the observed performance is reproducible under an independent training initialization. We have clarified the distinction between repeated evaluations and independent training runs in the revised manuscript.

Concern 3 (Weakness 1 & Question 1): The additional benefit of GRPO, particularly the proposed normalization and format-reward treatment, is therefore less firmly established than the benefit of specialist training overall. Is there evidence isolating the benefit of the (1/N_i) normalization or the window-specific format penalty?

Response 3: We thank the reviewer for suggesting a more focused analysis of the reward components. We conduct additional ablations on Window-count Normalization and Format Reward to quantify their respective contributions to downstream QA performance.

Window-count NormalizationFormat RewardMHTempODSHOverall₄
✓✓48.05 ± 0.3462.64 ± 0.4829.79 ± 1.9468.33 ± 0.0461.03 ± 0.26
×✓45.03 ± 0.2260.45 ± 0.8530.48 ± 1.5766.98 ± 0.5059.32 ± 0.16
✓×46.86 ± 0.7261.88 ± 1.1331.46 ± 1.8368.27 ± 0.1260.73 ± 0.29

Removing Window-count Normalization decreases Overall₄ F1 by 1.71 points. This component addresses a granularity mismatch in our training design: memory utility is evaluated jointly at the conversation level, whereas the Archiver generates memory entries independently at the window level. Specifically, we scale each window's normalized advantage by the inverse number of valid windows in its conversation (1/N_i), preventing conversations with more valid windows from contributing disproportionately to optimization.

Removing Format Reward decreases Overall₄ F1 by 0.30 points. This component constrains the output format of window-level memory entries, encouraging structurally consistent memory generation.

AuthorsOpenReview ↗

To Reviewer 66gM/#1 (2/3)

Concern 4 (Weakness 2 & Question 2): The retrieval ablations do not fully isolate evidence-based stopping and gap-aware refinement. Table 5 describes its standard comparator as a SimpleMem-style fixed-top-(k) strategy. However, SimpleMem explicitly includes adaptive or intent-aware retrieval planning, so the exact implementation, retrieval budget, and enabled features of this comparator need clarification. What exact SimpleMem implementation and configuration constitute the standard retrieval baseline in Table 5?

Response 4: We thank the reviewer for pointing out that the Standard retrieval baseline requires a more precise description, particularly regarding its relationship to SimpleMem.

Our Standard retrieval baseline follows the multi-view retrieval design of SimpleMem, which combines semantic, lexical, and symbolic retrieval. Specifically, an LLM prepares semantic subqueries, keywords, and structured retrieval information, which are used to retrieve candidates through three complementary operations:

Retrieval OperationQuery SourceCandidate Budget
Dense RetrievalLLM-decomposed semantic subqueriesTop-25
BM25 Keyword RetrievalExtracted keywordsTop-5
Symbolic RetrievalTemporal, location, and entity informationTop-5

These three retrieval operations constitute our Standard retrieval baseline in Table 5. The baseline uses fixed candidate budgets and does not include the subsequent evidence-gap-driven query refinement introduced in Tier 4.

We have clarified these implementation details to distinguish the retrieval components used in our baseline from the additional Gap-aware Planning mechanism evaluated in LocalMem.

Concern 5 (Weakness 2 & Question 2): In addition, progressing from Tier 3 to Tier 4 changes both the retrieval queries and the candidate budgets. The improvement in Table 6 therefore does not separately establish the value of gap-aware planning. A focused control that expands the same retrieval queries to the Tier-4 budget without generating a gap plan would help identify this contribution. How does Tier 4 compare with increasing the candidate budget while retaining the Tier-3 queries?

Response 5: We thank the reviewer for suggesting a controlled experiment to isolate the contribution of Gap-aware Planning. We conduct an additional comparison by first expanding the Tier-3 candidate budget without changing its original retrieval queries, and then enabling Gap-aware Planning while keeping the expanded Top-K settings unchanged.

Expand Top-KGap PlanMHTempODSHOverall₄Δ
××44.71 ± 0.2760.79 ± 0.5129.66 ± 0.9166.86 ± 0.2959.22 ± 0.04—
✓×48.10 ± 0.4362.48 ± 0.3828.52 ± 1.3767.86 ± 0.0460.67 ± 0.09+1.45
✓✓48.31 ± 0.2763.57 ± 0.8530.14 ± 0.5768.51 ± 0.3061.39 ± 0.15+2.17

Expanding the Tier-3 candidate budget alone improves Overall₄ F1 from 59.22 to 60.67 (+1.45 points). With the candidate budget held constant, enabling Gap-aware Planning further improves Overall₄ F1 to 61.39 (+0.72 points).

This controlled comparison isolates the additional contribution of Gap-aware Planning beyond simply increasing the retrieval budget. The results show that explicitly identifying unsupported information requirements and refining retrieval queries provides further gains over expanded fixed-query retrieval.

Concern 6 (Question 3): Please identify the cleaned evaluation split/version, any exclusions, the adversarial-question scoring procedure, and the validation procedure used to select checkpoints and retrieval settings.

Response 6: We thank the reviewer for requesting clarification of the evaluation splits, scoring rules, and model-selection protocol. We provide the benchmark details below. LocalMem is trained exclusively on the official Mem-α Train Split, without using LoCoMo10 or LongMemEval-S data for training.

BenchmarkEvaluation SplitConversationsQuestionsAdversarial Questions
LoCoMo10Full Evaluation Set101,986446
LongMemEval-SCleaned Evaluation Split500500—

For LongMemEval-S, we use the official cleaned evaluation split and do not apply additional sample filtering or exclusions beyond the benchmark-provided preprocessing.

For adversarial questions in LoCoMo10, we follow the benchmark's refusal-style scoring rule: a prediction containing "no information available" or "not mentioned" receives a score of 1. Otherwise, it receives a score of 0.

For model selection, we use validation performance with early stopping to select checkpoints. Retrieval Top-K values are based on the configurations of SimpleMem and related retrieval methods, while the number of retrieval tiers is determined through retrieval-depth ablations. Neither LoCoMo10 nor LongMemEval-S is used for training.

AuthorsOpenReview ↗

To Reviewer 66gM/#1 (3/3)

Concern 7 (Question 4): How can Tier 4 recover a previously searched but still unsupported information need under the prompt in Appendix A.8.4? Please provide an example including the original requirements, attempted requirements, returned gap plan, and newly retrieved evidence.

Response 7: We thank the reviewer for requesting a concrete example of how Tier 4 recovers previously attempted but unsupported information. We provide an actual runtime example from LoCoMo10 Sample 3, Q51, illustrating how Gap-aware Planning refines an existing retrieval requirement to recover missing evidence.

Question: "When did Nate get Tilly for Joanna?" Answer: 25 May, 2022.

The original information requirement submitted to Tier 3 is:

required_info = [
    "The date when Nate got Tilly for Joanna"
]

Tier 3 performs Semantic Top-25, Keyword Top-5, and Structured Top-5 retrieval, returning 31 memory entries after aggregation. Five entries mention Tilly, but all originate from a conversation on 2022-10-21. They establish that Tilly is a stuffed toy dog given to Joanna by Nate, but none contains the date of the gift. The Evidence-sufficiency Judge therefore determines that the retrieved evidence does not satisfy the original requirement.

The Gap Planner receives the original requirement, the attempted requirement, and the retrieved evidence. It identifies the missing information as follows:

missing_information = [
    "The specific date when Nate got the stuffed animal Tilly for Joanna"
]

Based on the retrieved object description, the planner refines "Tilly" into "the stuffed animal Tilly", making the entity description more explicit. Using this refined requirement, Tier 4 performs Semantic Top-50 and Keyword Top-25 retrieval and recovers the following memory entry:

"On 25 May 2022 at 3:00 pm, Nate sent Joanna a stuffed animal as a gift to remind her of the good vibes."

The associated timestamp, 2022-05-25T15:00:00, provides the requested answer.

Retrieval MeasureOriginal RequirementGap-refined Requirement
Semantic Retrieval Rank (common Top-50 ranking)39 / 503 / 50
Cosine Similarity0.78690.8755
Keyword RetrievalNot retrievedNot retrieved
Within Top-25NoYes

The original query's rank is reported in the same Top-50 ranking space for comparison, although it falls outside Tier 3's actual Top-25 retrieval budget. After query refinement, the relevant entry moves from rank 39 to rank 3 and enters the effective candidate range.

This example illustrates the complementary roles of the two components: the Evidence-sufficiency Judge identifies that the retrieved evidence does not cover the original requirement, while the Gap Planner identifies the missing information and refines the retrieval query to recover the relevant memory.

Concern 8 (Ethics Concerns): The privacy motivation should remain narrowly stated: local memory construction avoids transmitting complete raw histories, but retrieved memory entries can still contain sensitive information when sent to an external judge or Answer Model. The architecture does not by itself establish a formal privacy guarantee.

Response 8: We thank the reviewer for pointing out the need to narrow our privacy claims. We agree that the privacy benefit of LocalMem should be described in terms of local processing of complete conversation histories and selective memory transfer, rather than as a formal privacy guarantee.

Specifically, complete raw conversation histories are processed and retained locally rather than being transferred to external models. Only selected memory entries retrieved from the local archive for the current query are transmitted to the external Judge or Answer Model.

We acknowledge that these selected memory entries may still contain sensitive information. Therefore, we characterize the privacy benefit as avoiding the transfer of complete raw conversation histories, without claiming end-to-end privacy protection or a formal privacy guarantee.

We express our sincere gratitude to the reviewer for dedicating time to review our paper. We have provided comprehensive responses to all the concerns. As the discussion deadline looms within 2 days, we would like to inquire if our responses have adequately addressed your questions. We are more than willing to address any concerns and ensure a comprehensive resolution. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

Thank you and the reviewers for your time and constructive feedback. We summarize the key strengths, our responses, and the discussion outcomes below.

LocalMem addresses long-term memory challenges by keeping raw interaction histories local and reducing memory-management costs. Specialized 4B models handle memory construction and retrieval, while larger external LLMs perform evidence assessment, reasoning, and answer generation.

Methodologically, LocalMem introduces two complementary mechanisms. For memory construction, window-count-normalized GRPO aligns conversation-level downstream QA utility with independently generated window-level memory entries, while format rewards encourage structurally consistent outputs. For retrieval, a sufficiency judge assesses whether the retrieved evidence covers the original query's information requirements. When evidence is insufficient, the Memory Enquirer identifies missing information and progressively acquires targeted evidence.

Strengths Recognized Across Reviewers

  • Practical Motivation: Improving interactive agents' long-term memory while reducing costs is relevant and practical (ixFa).
  • Modular Architecture: Separating memory management from answer generation enables reuse with different downstream models (4n6D, 66gM).
  • Independent Construction: Window-to-entry generation avoids sequential editing and supports parallel processing (4n6D, 66gM).
  • Progressive Retrieval: Evidence-sufficiency-guided retrieval enables adaptive acquisition without unnecessary planning (4n6D, 66gM).
  • Controlled Evaluation: Matched-model comparisons and ablations validate individual components (4n6D, 66gM).
  • Model Efficiency: Specialized 4B memory models achieve comparable retrieval performance with lower latency (4n6D, 66gM).

Reviewer 66gM (Rating: 3; Confidence: 3)

  • Strengths: The reviewer praised LocalMem's "practical and modular design," parallel construction, memory/answer separation, comparisons with a shared Answer Model, and component ablations.
  • Concerns: More evidence on GRPO/rewards, training variability, Gap-aware Planning, and evaluation and privacy details.
  • Our Response: GRPO improves Overall₄ F1 by 0.49 points and Multi-Hop F1 by 1.34 points, while reducing the memory-to-dialog token ratio from 72.70% to 70.08%. Ablations attribute 1.71 and 0.30 Overall₄ F1 points to Window-count Normalization and Format Reward. We clarified that checkpoint evaluations are not independent runs and added a seed-1 run (61.06 Overall₄ F1 vs. 61.03). We specified baseline budgets, showed a 0.72-point Gap-aware Planning gain at a fixed expanded Top-K budget, clarified evaluation protocols, and added a retrieval example. Raw histories stay local; selected entries may be shared externally.

Reviewer 4n6D (Rating Raised from 2 to 3; Confidence: 3)

  • Strengths: The reviewer called LocalMem a "coherent and practically motivated learned memory pipeline" and praised its parallel construction, guided retrieval, and controlled validation.
  • Concerns: Clarification of the all-compact configuration and larger Judge/Answer models, training variability, and matched-baseline evaluation on more tasks.
  • Our Response: The all-4B configuration achieves 55.58 Overall₄ F1, while 4B memory models with 122B Judge/Answer models achieve 61.28. We clarified the complementary roles of local memory management and downstream reasoning and added a seed-1 run. The 4B Enquirer achieves 61.3 ± 0.2 Overall₄ F1 with 4.2-second latency, comparable to the 122B Progressive Enquirer (61.2 ± 0.2 F1, 6.7 seconds). We added matched-model evaluations on Mem-Gallery and Mem2ActBench.
  • Reviewer Feedback: The reviewer thanked us and raised the rating from 2 to 3.

Reviewer ixFa (Rating: 2; Confidence: 3)

  • Strengths: The reviewer found the problem "relevant" to improving LLM agents' long-term memory while reducing costs, and the paper "generally clearly written."
  • Concerns: Clarification of our contributions over prior memory/adaptive-retrieval work, more GRPO evidence, broader evaluation, and explanation of cross-window references and changing facts.
  • Our Response: We clarified that LocalMem aligns conversation-level QA utility with independent window-level generation and guides retrieval using evidence sufficiency and missing information. Window-count Normalization adds 1.71 Overall₄ F1 points; Gap-aware Planning adds 0.72 at a fixed expanded Top-K budget. We added Mem2ActBench results. Adjacent windows overlap by two turns to preserve context; timestamps help downstream models interpret changing facts.
  • Reviewer Feedback: The reviewer said the added experiments strengthened the paper's soundness but maintained the original rating. In our rebuttal, we further clarified LocalMem’s methodological contributions and its distinctions from prior work.

We hope this clarifies LocalMem's contributions and rebuttal evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.