acceptodds
Under review as a conference paper at ICLR 2027

Teaching LLMs to Remember: Parametric Memory via Personalized Adapter Weights

Abstract

Long-term personalization requires AI assistants to recall personal facts across sessions. Existing long-term memory systems mostly rely on token memory: facts are written as text and retrieved by searching among external embeddings. Although this design is practical and easy to maintain, retrieval is decoupled from answer utility, and the memory store grows with user history. Motivated by these limitations and the way LLMs compress knowledge into parameters, we explore whether personal memory can instead be represented at the parameter level. In this paper, we introduce **ParaMem**, a parametric memory framework for personal LLM assistants, where a user’s history is compressed into a compact per-user LoRA adapter and recalled by generating explicit memory contexts from the learned weights. This formulation shifts memory retrieval from external search to learned, personalized generation, while introducing two distinctive challenges: parametric recall must cover sufficient encoded history for a given query, and the generated memory should actually help answer the query rather than merely be related to it. To address recall coverage, we propose *ParamRecall*, an inference-time strategy that combines input-side query probing with output-side memory sampling, providing a parametric counterpart to top-K retrieval in token-memory systems. To study answer helpfulness, we further propose *memory-aware preference learning* and show that the quality of the negative construction is critical: different-subject hard negatives, which contrast memories about different people sharing the same fact type, provide a meaningful signal for distinguishing helpful memories from plausible but insufficient ones. Finally, we show that parametric and token memory are complementary: grounding the adapter's recalled memories in a stored fact bank corrects attribution errors and yields the best overall system. Experiments on real-world benchmarks show that ParaMem outperforms strong token-memory baselines across most question types, suggesting that parameter space is a promising design direction for long-term memory in AI assistants.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parametric Memory, Personalized LLMs, Long-term Conversational Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aSrzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ParaMem, a parametric memory framework for personal LLM assistants that compresses each user's history into a per-user LoRA adapter and recalls memory by generating explicit memory contexts from the learned weights, instead of retrieving from an external store. To improve recall coverage, the authors introduce ParamRecall, which combines query probing with memory sampling at inference time. To improve answer helpfulness, they apply memory-aware preference learning with different-subject hard negatives. They further show that grounding the generated memories in a stored fact bank corrects attribution errors, and report that ParaMem outperforms token-memory baselines on most question types in real-world benchmarks.

Critical strengths
  1. The problem is interesting: the paper studies how to parameterize user memory while still allowing the generated memories to be inspected.
  2. The experimental comparison is fair, with consistent settings across methods.
  3. The analysis is comprehensive and shows the trade-off between storage and accuracy.
  4. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The authors conduct experiments only on LoCoMo, which makes it difficult to assess the generalization ability of the proposed method.
  2. According to Table 4, ParaMem + grounding performs very close to No-adapter gen + grounding, suggesting that the contribution of the adapter is very limited.
  3. In Table 3, the differences among most results are small. Could the authors report results from multiple runs, or provide the mean and standard deviation, to allow a more reliable comparison?
  4. The evaluation metrics are relatively limited. The authors could add evaluation results based on LLM-as-a-judge.
Questions for authors

please refer to the weakness section

4 comments

We thank the reviewer for the thoughtful feedback. We believe that our paper will be much stronger thanks to the reviewer's efforts.

The authors conduct experiments only on LoCoMo, which makes it difficult to assess the generalization ability of the proposed method.

We thank the reviewer for this suggestion and have added a second benchmark, PersonaMem (32k split) [1]. It contains 589 multiple-choice questions over 37 user histories and covers 7 question types, such as preference evolution, recommendation, and recall of shared facts.

  • The format differs from LoCoMo: user statements instead of factual questions, and multiple-choice instead of free-form answers.
  • We applied ParaMem unchanged: the same synthesis prompts, adapter hyperparameters, and ParamRecall settings.
  • All systems share one teacher/ingestion model, Qwen3.5-397B, and the same Qwen2.5-3B answerer and prompt.
  • Within each history, all systems see the same prefix: the conversation up to that history's earliest question.
systemaccuracycontext tokens/q
No memory0.4260
Full context0.44122,743
Mem00.46357
SimpleMem0.47447
Summarization0.48485
A-Mem (k=10)0.49616,677
ParaMem-context0.49115
+ParamRecall0.49760
+ParamRecall+grounding0.508222

The main findings transfer.

  • Parametric memory alone matches the strongest token-memory baseline. ParaMem+ParamRecall, which uses no stored text at all, matches A-Mem while giving the answering model a context that is two orders of magnitude shorter.
  • It also outperforms full context and Mem0, and grounding improves it further.

We will add these results as a new appendix section.

According to Table 4, ParaMem + grounding performs very close to No-adapter gen + grounding, suggesting that the contribution of the adapter is very limited.

We thank the reviewer for the sharp observation. It exposed a flaw in how we set up Table 4: the comparison there is not fully controlled. Due to an oversight, the no-adapter control drew twice as many memory samples per question as ParaMem: kq=4k_q=4 probes ×\times ks=3k_s=3 samples, vs. kq=2k_q=2 probes ×\times ks=3k_s=3 samples. Once the sampling budget is matched, the adapter's contribution is clear and distinct.

Controlled comparison. We re-ran the control with exactly ParaMem's sampling budget: the same number of probes and samples and the same sampling settings. The only differences are the absence of the adapter and a HyDE-style prompt [2] asking the backbone to write a plausible memory.

(a) Table 4 setting. With the sample-matched control, ParaMem + grounding distinctly outperforms no-adapter gen + grounding on both backbones, where both use the same sampling budget (kq=2, ks=3).

(b) Matched fact budget. To isolate the adapter's contribution, every method receives exactly K facts, all retrieved by BM25 from the same fact store; the methods differ only in the BM25 query: the question (BM25), the untrained backbone's generated memories (no-adapter), or the adapter's recalled memories (ParaMem). For the latter two, the sampling setups are the same as before (kq=2, ks=3), and each generated memory is replaced by its top-1 BM25 fact; these facts come first, and BM25 (question) fills any remaining slots. F1 overall:

K3B BM253B no-adapter3B ParaMem8B BM258B no-adapter8B ParaMem
10.3280.3240.3890.3850.3730.464
20.3500.3600.4090.4100.4040.478
30.3670.3780.4210.4260.4300.488
50.3990.4150.4440.4460.4560.491
100.4240.4420.4590.4590.4760.495
150.4320.4490.4560.4740.4860.495
  • The adapter retrieves better facts.
    • ParaMem beats the no-adapter control at every budget, on both backbones. The difference is significant for every K≤10K\leq 10. At K=15K=15, BM25(question) supplies most of the context, and the gap narrows to a non-significant margin.
    • The gap is largest when the budget is tight. At small K, most of the K facts are retrieved with each method's own query (the question for BM25, the generated memories for no-adapter and ParaMem), so the quality of that query largely determines the context. At large K, the remaining slots are filled with the same BM25(question) facts for every method, so the contexts converge.
  • The adapter's few facts are the most valuable ones. At large budgets, BM25 fills most of the context. Still, replacing just a couple of BM25 facts with adapter-recalled facts gives a clear gain over pure BM25, larger than the gain from (untrained) backbone-generated memories.

We will replace the Table 4 control with the sample-matched one, add the matched-budget table to the main text, and rewrite the discussion. We thank the reviewer again for prompting this correction.

In Table 3, the differences among most results are small. Could the authors report results from multiple runs, or provide the mean and standard deviation, to allow a more reliable comparison?

We thank the reviewer for this suggestion. Running Table 3 with multiple seeds showed that the submitted DPO variants were within run-to-run noise of SFT. We traced this to one design choice and revised it.

What changed. Only the source of the rejected memories. Everything else is unchanged: the DPO objective, the SFT initialization, β, the trainer, the training queries, and the three negative types. Preference learning is also a separate ablation, so none of the main results are affected.

Why. The submitted negatives were memories mined from the user's fact store. Before DPO starts, the SFT adapter already assigns these negatives a much lower likelihood than the correct memory. Most of the DPO update therefore lowers the probability of memories the adapter rarely samples, and has little effect on what it recalls at inference. This is consistent with prior findings that DPO works better when the rejected responses are sampled from the model being trained [3,4]. We now take negatives from the adapter itself: it recalls memories for paraphrased training queries, and recalls that ground to a wrong fact become the rejected memories, each assigned to one of the three types by the fact it grounds to. All three types use the same hyperparameter grid setup, with configurations selected by two-fold cross-user selection.

SystemAvgC1C2C3C4C5
SFT (no preference learning)0.348 ± 0.0000.262 ± 0.0110.334 ± 0.0060.197 ± 0.0260.378 ± 0.0060.387 ± 0.008
DPO + BM25-hard negatives0.350 ± 0.0020.262 ± 0.0080.328 ± 0.0170.197 ± 0.0110.381 ± 0.0050.397 ± 0.004
DPO + cross-fact negatives0.351 ± 0.0040.276 ± 0.0050.328 ± 0.0120.211 ± 0.0100.382 ± 0.0090.386 ± 0.011
DPO + different-subject negatives0.354 ± 0.0040.271 ± 0.0070.339 ± 0.0010.210 ± 0.0170.378 ± 0.0050.404 ± 0.007
Qwen2.5-3B, DeepSeek teacher; mean ± sd over 3 inference seeds.
  • All three negative types now improve over SFT. Different-subject negatives remain the best, consistent with the paper's claim.
  • The gains are modest by design. DPO only refines which stored memory the adapter recalls. We keep the preference update small so that it does not overwrite the facts learned in SFT.

We will revise Table 3 with new results in the next version.

The evaluation metrics are relatively limited. The authors could add evaluation results based on LLM-as-a-judge.

We thank the reviewer for the thoughtful suggestion. We re-evaluate the paper's exact outputs with three additional measures:

  • the Mem0 LLM-judge prompt [5], run with two judges different from the teacher, to avoid self-preference: Qwen3.5-397B (J-Q) and gpt-oss-120b (J-G);
  • BLEU-1;
  • SBERT similarity.
Qwen2.5-3BJ-QJ-GBLEU-1SBERT
FullContext0.2670.2700.2020.375
Summarization0.4400.4260.2870.515
LangMem0.2650.2650.1860.422
Mem00.3290.3250.2180.431
ReadAgent0.3480.3440.2490.451
SimpleMem0.4250.4180.2680.488
A-Mem0.3820.3820.3010.529
Param A-Mem0.2800.2840.2420.476
ParaMem-direct0.3670.3510.2500.485
ParaMem-context0.4410.4400.2690.510
+ParamRecall0.4530.4460.2890.527
+ParamRecall+grounding0.5930.5820.3870.622
Llama-3.1-8BJ-QJ-GBLEU-1SBERT
FullContext0.3430.3450.2590.455
Summarization0.5210.5060.3380.572
LangMem0.2320.2290.2050.304
Mem00.3370.3350.2460.454
ReadAgent0.4340.4340.3020.515
SimpleMem0.4090.4010.2780.481
A-Mem0.4470.4380.3150.542
Param A-Mem0.2800.2790.2120.432
ParaMem-direct0.3860.3740.2610.497
ParaMem-context0.4860.4660.2930.547
+ParamRecall0.4950.4860.2990.554
+ParamRecall+grounding0.6090.5940.3890.626
  • The main conclusion does not depend on the metric. Full ParaMem is best on every metric and on both backbones, and the two independent judges agree closely.
  • Parametric recall alone is competitive. Without any fact store, ParaMem+ParamRecall outperforms every token-memory baseline under both LLM judges on Qwen2.5-3B, and is second only to Summarization on Llama-3.1-8B under the LLM judge.

We will add these columns to Tables 1 and 4.

[1] Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." COLM 2025.

[2] Gao, Luyu, et al. "Precise zero-shot dense retrieval without relevance labels." ACL 2023.

[3] Tajwar, Fahim, et al. "Preference fine-tuning of llms should leverage suboptimal, on-policy data." ICML 2024.

[4] Xu, Shusheng, et al. "Is dpo superior to ppo for llm alignment? a comprehensive study." ICML 2024.

[5] Chhikara, Prateek, et al. "Mem0: Building production-ready ai agents with scalable long-term memory." arXiv preprint 2025.

Thank you for the response. After reading the results and clarifications, most of my concerns have been resolved. I will raise my score. Thanks to the authors' efforts in the rebuttal phase.

Reviewer_UYgUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies long-term memory for personal AI assistants and proposes ParaMem, a parametric memory framework where a user’s conversation history is compressed into a per-user LoRA adapter, and relevant memories are recalled by generating them from the adapter.Experiments on LoCoMo show that personal memory can be encoded into user-specific parameters and effectively recalled by ParaMem, with substantial gains over existing parametric baselines.

Critical strengths
  1. The paper demonstrates that parametric memory can be used effectively in LLM-based personal AI assistants. Its gains over both token-memory and parametric-memory baselines on LoCoMo suggest that parametric memory is a promising component for building personalized memory systems.

  2. The paper provides a careful breakdown of the problems that arise when parametric memory is applied naively. The components proposed to address them, namely grounded fact extraction from conversation history, improved recall coverage, and enhanced answer helpfulness, are likely to benefit future work on parametric memory.

Critical weaknesses
  1. ParaMem appears to operate under a considerably larger compute budget than both the token-memory and parametric-memory baselines. ParamRecall generates kq×ksk_q \times k_s memory contexts per question (multiple query paraphrases ×\times multiple samples each), which is likely far more expensive than the single retrieval call used by token-memory baselines. Moreover, ParaMem is effectively a hybrid system: it grounds recalled memories in a teacher-extracted fact store via BM25. It therefore requires both per-user adapter training and a token-level fact store, incurring the costs of both paradigms. Since some accuracy gain would naturally be expected from this additional computation, it is difficult to tell how much of the improvement comes from the proposed approach itself.

  2. The paper reports only token-level F1 on LoCoMo, whereas several prior works (e.g., A-Mem [1], Mem0 [2], Memory-R1 [3], REMem [4]) also report BLEU-1, embedding-based similarity (SBERT), and/or an LLM-as-a-judge score. Adding a semantic metric, ideally an LLM-as-a-judge score with a shared answering prompt across all methods, would make the results more convincing.

References

[1] W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang, “A-Mem: Agentic Memory for LLM Agents,” NeurIPS, 2025.

[2] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav, “Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory,” arXiv:2504.19413, 2025.

[3] S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, H. Schütze, V. Tresp, and Y. Ma, “Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning,” ACL, 2026.

[4] Y. Shu, S. P. Jonnalagedda, X. Gao, B. Jiménez Gutiérrez, W. Qi, K. Das, H. Sun, and Y. Su, “REMem: Reasoning with Episodic Memory in Language Agent,” ICLR, 2026.

Questions for authors
  1. How much additional cost does ParaMem incur relative to the baselines in practice, for example in terms of generated tokens or wall-clock time per question?

  2. Could the authors add a semantic metric, ideally an LLM-as-a-judge score with a shared answering prompt across all methods?

3 comments

We thank the reviewer for the thoughtful feedback. We believe that our paper will be much stronger thanks to the reviewer's efforts.

ParaMem appears to operate under a considerably larger compute budget than both the token-memory and parametric-memory baselines. ParamRecall generates kq×ksk_q\times k_s memory contexts per question (multiple query paraphrases ×\times multiple samples each), which is likely far more expensive than the single retrieval call used by token-memory baselines.

How much additional cost does ParaMem incur relative to the baselines in practice, for example in terms of generated tokens or wall-clock time per question?

We respectfully disagree that ParaMem is far more expensive in practice, and we thank the reviewer for asking us to quantify it. We measured the online cost of all systems with the same setup (Qwen2.5-3B, batch size 1, 200 random LoCoMo questions, kq=2, ks=3, K=10).

systemlatency (s)LLM callsprompt tokgen tok
Facts+BM250.2312577
Mem00.83335428
A-Mem1.0928,84017
Summarization1.4631,59851
SimpleMem1.64387061
ParaMem-context0.71215822
+ParamRecall1.27840888
+ParamRecall+grounding1.26855186
No-adapter gen + grounding1.508647155

ParamRecall does issue more calls than a single retrieval. However, each call is a short generation, and the six memory samples are generated in one batched round, so the wall-clock overhead is small. ParaMem's latency is comparable to the token-memory baselines: it is slower than Mem0 and A-Mem and faster than Summarization and SimpleMem. ParaMem and token memory spend their cost differently:

  • Token memory pays for long retrieved contexts that the answering model must read.
  • ParaMem pays for a few short, batched generations, and the answering model reads a compact context.

Generating kq×ksk_q\times k_s memory contexts does not give ParaMem a larger retrieval budget. ParamRecall is the parametric counterpart of top-K retrieval: kq×ksk_q\times k_s bounds the number of recalled memories, just as K bounds the number of retrieved units.

  • ParaMem's budget is smaller. Its 6 samples deduplicate to only about 2–3 distinct memories per question.
  • The baselines retrieve more. Token-memory baselines retrieve the top-10 units, and A-Mem further expands each retrieved note with its linked neighbors.

ParaMem therefore hands the answering model fewer memory units than the baselines do.

Multiple samples are needed only because the adapter is trained to emit one memory per generation. A meaningful future direction is to train it to emit the whole set of relevant memories in a single generation. That would turn ParamRecall's multiple calls into a single call. We will mention this in the revision.

Moreover, ParaMem is effectively a hybrid system: it grounds recalled memories in a teacher-extracted fact store via BM25. It therefore requires both per-user adapter training and a token-level fact store, incurring the costs of both paradigms. Since some accuracy gain would naturally be expected from this additional computation, it is difficult to tell how much of the improvement comes from the proposed approach itself.

We respectfully disagree that the source of the improvement is hard to tell. The paper's design isolates the two retrieval channels:

  • Parametric channel only: ParaMem+ParamRecall (Table 1) retrieves exclusively through the LoRA adapter, with no fact store at all.
  • Token channel only: Facts+BM25 (Table 4) retrieves from the same fact store with BM25, with no adapter.
  • Both channels: ParaMem+grounding (Tables 1 and 4) combines the two.

Each single channel is competitive on its own, and combining them performs best on both backbones. The two kinds of memory are therefore complementary, and the hybrid's gain is not a by-product of one of them.

The new matched-budget study (reply to Reviewer KyvT) makes this isolation tighter. Every method receives exactly the same number of facts from the same store:

  • Against the token channel alone: replacing a few BM25 facts with adapter-recalled facts distinctly improves over pure BM25.
  • Against a sample-matched control: the control's backbone generates as many memory samples as ParaMem, and those generations are grounded in exactly the same way. ParaMem is still clearly better.

Since the sampling budget, the number of facts, and the fact store are all matched, the remaining difference is the adapter.

On cost, we agree that the grounded variant needs both an adapter and a fact store. The fact store is the same kind of compact text store that token-memory baselines already maintain, and the parametric-only variant needs none. The offline cost is one-time:

  • Fact extraction and data synthesis take one teacher pass per session, comparable to the LLM extraction calls that token-memory baselines make at write time.
  • Adapter training runs once per user, asynchronously, like a write path, so it adds no online latency.

We will add this discussion to the paper.

The paper reports only token-level F1 on LoCoMo, whereas several prior works (e.g., A-Mem [1], Mem0 [2], Memory-R1 [3], REMem [4]) also report BLEU-1, embedding-based similarity (SBERT), and/or an LLM-as-a-judge score...

Could the authors add a semantic metric, ideally an LLM-as-a-judge score with a shared answering prompt across all methods?

We thank the reviewer for the thoughtful suggestion. All systems in the paper already share the same backbone and the same category-aware answering prompt. We re-evaluate the paper's exact outputs with three additional measures:

  • the Mem0 LLM-judge prompt [1], run with two judges different from the teacher, to avoid self-preference: Qwen3.5-397B (J-Q) and gpt-oss-120b (J-G);
  • BLEU-1;
  • SBERT similarity.
Qwen2.5-3BJ-QJ-GBLEU-1SBERT
FullContext0.2670.2700.2020.375
Summarization0.4400.4260.2870.515
LangMem0.2650.2650.1860.422
Mem00.3290.3250.2180.431
ReadAgent0.3480.3440.2490.451
SimpleMem0.4250.4180.2680.488
A-Mem0.3820.3820.3010.529
Param A-Mem0.2800.2840.2420.476
ParaMem-direct0.3670.3510.2500.485
ParaMem-context0.4410.4400.2690.510
+ParamRecall0.4530.4460.2890.527
+ParamRecall+grounding0.5930.5820.3870.622
Llama-3.1-8BJ-QJ-GBLEU-1SBERT
FullContext0.3430.3450.2590.455
Summarization0.5210.5060.3380.572
LangMem0.2320.2290.2050.304
Mem00.3370.3350.2460.454
ReadAgent0.4340.4340.3020.515
SimpleMem0.4090.4010.2780.481
A-Mem0.4470.4380.3150.542
Param A-Mem0.2800.2790.2120.432
ParaMem-direct0.3860.3740.2610.497
ParaMem-context0.4860.4660.2930.547
+ParamRecall0.4950.4860.2990.554
+ParamRecall+grounding0.6090.5940.3890.626
  • The main conclusion does not depend on the metric. Full ParaMem is best on every metric and on both backbones, and the two independent judges agree closely.
  • Parametric recall alone is competitive. Without any fact store, ParaMem+ParamRecall outperforms every token-memory baseline under both LLM judges on Qwen2.5-3B, and is second only to Summarization on Llama-3.1-8B.

We will add these columns to Tables 1 and 4.

[1] Chhikara, Prateek, et al. "Mem0: Building production-ready ai agents with scalable long-term memory." arXiv preprint 2025.

I appreciate the latency measurements, which help clarify the computational cost and the contribution of the parametric adapter. The additional semantic evaluation results also adequately address my concern regarding the reliance on token-level F1.

While the end-to-end cost-efficiency, particularly when accounting for per-user adapter training and memory updates, remains an open question, I recognize that the authors have acknowledged these limitations and discussed the relevant trade-offs in the revised manuscript.

Overall, the rebuttal has addressed most of my concerns and strengthened the empirical support for the paper's contributions. I will raise my rating.

Reviewer_KyvTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to train parametric memory (which they term ParaMem) for personal facts recall. For each user, its ParaMem is essentially a LoRA adapter trained on facts collected from the user's conversations. There two types of training objective: 1. direct supervision on the answer of a query; 2. supervision on the query-related context. The adapter is trained to fit these two objectives conditioned on different system prompts. When deploy this ParaMem adapter, the authors propose a method ParamRecall, which paraphrases the query and samples multiple query-related contexts. The backbone LLM takes these contexts and the query as the input and generate an answer. To reduce hallucinations, the authors add a verification layer called grouding, which replaces the output of ParaMem with the closest fact from a fact store. To make sure the generated context is helpful, they use DPO to train the adapter on useful and non-helpful contexts after the LoRA tuning. Evaluations show that ParaMem-based methods achieves better F1 score on facts retrievel.

Critical strengths

The idea of one adapter per user is interesting. The verfication layer of grouding is a reasonable design.

Critical weaknesses

The advantage of parametrized memory is not suffciently verified. From Table 4, it seems like the performance improvement comes from the grouding component and no-adapter gen + grounding achieves competing performance. Moreover no-adapter gen + grounding seems more efficient in terms of space and time (not well discussed in the paper).

Questions for authors
  1. Is there a inference time comparason to all baselines?
  2. Right now the adapter takes a fixed space complexity, which also limits the memory capacity. Is there a comprehensive study on how adapter size scales with the memory capacity? How does this scaling compare to baseline methods?
4 comments

We thank the reviewer for the thoughtful feedback. We believe that our paper will be much stronger thanks to the reviewer's efforts.

The advantage of parametrized memory is not suffciently verified. From Table 4, it seems like the performance improvement comes from the grouding component and no-adapter gen + grounding achieves competing performance.

We thank the reviewer for the sharp observation. It exposed a flaw in how we set up Table 4: the comparison there is not fully controlled. Due to an oversight, the no-adapter control drew twice as many memory samples per question as ParaMem: kq=4k_q=4 probes ×\times ks=3k_s=3 samples, vs. kq=2k_q=2 probes ×\times ks=3k_s=3 samples. Once the sampling budget is matched, the adapter's contribution is clear and distinct.

Controlled comparison. We re-ran the control with exactly ParaMem's sampling budget: the same number of probes and samples and the same sampling settings. The only differences are the absence of the adapter and a HyDE-style prompt [1] asking the backbone to write a plausible memory.

(a) Table 4 setting. With the sample-matched control, ParaMem + grounding distinctly outperforms no-adapter gen + grounding on both backbones, where both use the same sampling budget (kq=2, ks=3).

(b) Matched fact budget. To isolate the adapter's contribution, every method receives exactly K facts, all retrieved by BM25 from the same fact store; the methods differ only in the BM25 query: the question (BM25), the untrained backbone's generated memories (no-adapter), or the adapter's recalled memories (ParaMem). For the latter two, the sampling setups are the same as before (kq=2, ks=3), and each generated memory is replaced by its top-1 BM25 fact; these facts come first, and BM25 (question) fills any remaining slots. F1 overall:

K3B BM253B no-adapter3B ParaMem8B BM258B no-adapter8B ParaMem
10.3280.3240.3890.3850.3730.464
20.3500.3600.4090.4100.4040.478
30.3670.3780.4210.4260.4300.488
50.3990.4150.4440.4460.4560.491
100.4240.4420.4590.4590.4760.495
150.4320.4490.4560.4740.4860.495
  • The adapter retrieves better facts.
    • ParaMem beats the no-adapter control at every budget, on both backbones. The difference is significant for every K≤10K\leq 10. At K=15K=15, BM25(question) supplies most of the context, and the gap narrows to a moderate margin.
    • The gap is largest when the budget is tight. At small K, most of the K facts are retrieved with each method's own query (the question for BM25, the generated memories for no-adapter and ParaMem), so the quality of that query largely determines the context. At large K, the remaining slots are filled with the same BM25(question) facts for every method, so the contexts converge.
  • The adapter's few facts are the most valuable ones. At large budgets, BM25 fills most of the context. Still, replacing just a couple of BM25 facts with adapter-recalled facts gives a distinct gain over pure BM25, which (untrained) backbone-generated memories do not match.

We will replace the Table 4 control with the sample-matched one, add the matched-budget table to the main text, and rewrite the discussion. We thank the reviewer again for prompting this correction.

Is there a inference time comparason to all baselines?

Yes, we thank the reviewer for the request. We measured all systems with the same setup (Qwen2.5-3B, batch size 1, 200 random LoCoMo questions, kq=2, ks=3, K=10).

systemlatency (s)LLM callsprompt tokgen tok
Facts+BM250.2312577
Mem00.83335428
A-Mem1.0928,84017
Summarization1.4631,59851
SimpleMem1.64387061
ParaMem-context0.71215822
+ParamRecall1.27840888
+ParamRecall+grounding1.26855186
No-adapter gen + grounding1.508647155

ParaMem's latency is comparable to the token-memory baselines: it is slower than Mem0 and A-Mem and faster than Summarization and SimpleMem. ParaMem and token memory spend their cost differently:

  • Token memory pays for long retrieved contexts that the answering model must read.
  • ParaMem pays for a few short, batched generations, and the answering model reads a compact context.

The table also addresses the reviewer's remark that no-adapter gen + grounding seems more efficient: it is slower than ParaMem even with a matched sampling budget, since the base model writes longer hypothetical memories.

Right now the adapter takes a fixed space complexity, which also limits the memory capacity. Is there a comprehensive study on how adapter size scales with the memory capacity? How does this scaling compare to baseline methods?

We thank the reviewer for raising this important question. For ParaMem, adapter size is a design choice and does not grow with the conversation:

  • History grows moderately: new memories are written into the same adapter, so storage and per-query inference cost stay constant.
  • History grows substantially: capacity is raised by increasing the LoRA rank r or the number of adapted layers l.
  • Number of users grows: we either add adapters (per-user) or share/group adapters across users, as shown in Table 5.

Storage therefore grows in discrete steps that we control, rather than with every new turn.

In addition, we already provided storage cost results in Table 5. Table 5 records the performance and storage cost under different LoRA ranks, adapter layers, and user-grouping for adapters:

  • Storage stays modest, and within each sharing scheme, F1 rises monotonically with storage.
  • A single adapter shared by all 10 users already matches the performance of the best token memory baseline.

Comparison to token-memory baselines. Their storage grows linearly with conversation length: every session adds text records and embeddings (Mem0, A-Mem, SimpleMem), summaries (Summarization), or raw history (FullContext). Their retrieval context grows as well. On LoCoMo, A-Mem already reads 8.8k prompt tokens per question; on the longer PersonaMem histories it reads 16.7k, and full context reaches 22.7k.

We note that at LoCoMo scale, the absolute footprint of compact token stores is smaller than that of per-user adapters. The difference is in how each scales: token memory grows with every conversation, while ParaMem stays fixed until we choose to enlarge it. We will add this discussion next to Table 5.

[1] Gao, Luyu, et al. "Precise zero-shot dense retrieval without relevance labels." ACL 2023.

Thanks for the response and the new experiments. Is the sample-matched version of table 4 ready? I just checked the paper and seems like the paper has not been revised.

Thank you for the reminder. We have uploaded the revised paper, with all changes highlighted in blue:

  • Sample-matched Table 4. The control in Table 4 now uses exactly ParaMem's sampling budget, and the discussion in Appendix B.2 is rewritten accordingly.
  • Matched fact budget. The new Table 5 demonstrates the results of controls with different matched budgets.
  • Inference cost. The latency comparison with all baselines is in Table 9 and Appendix B.6.
  • Capacity scaling. The discussion of how adapter size scales, compared with token memory, is placed in Appendix B.5.

We hope these address your concerns, and we would be glad to answer any further questions.

Comments on the paper

1 comment

We have uploaded the revised paper. We thank all reviewers again for the thoughtful comments. The revision implements the changes promised in our responses; all changes are highlighted in blue.

  • Sample-matched no-adapter control updated in Table 4, and the matched fact-budget study in Table 5 (in response to reviewers KyvT, UYgU, aSrz).
  • Inference-cost comparison with all baselines, and the offline-cost discussion in Appendix B.6 (in response to reviewers KyvT, UYgU).
  • Capacity-scaling discussion added to Appendix B.5 (In response to reviewer KyvT).
  • LLM-as-a-judge (two new models), BLEU-1 and SBERT results for all systems in Table 6 (in response to reviewers UYgU, aSrz).
  • PersonaMem results (in response to reviewer aSrz).
  • Table 3 with on-policy negatives, mean ±\pm sd over seeds; Section 2.4, Appendix. A.4 and B.1 updated accordingly (in response to reviewer aSrz).

Thank you again for engaging with our work. We are happy to discuss any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.