- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ParaMem, a parametric memory framework for personal LLM assistants that compresses each user's history into a per-user LoRA adapter and recalls memory by generating explicit memory contexts from the learned weights, instead of retrieving from an external store. To improve recall coverage, the authors introduce ParamRecall, which combines query probing with memory sampling at inference time. To improve answer helpfulness, they apply memory-aware preference learning with different-subject hard negatives. They further show that grounding the generated memories in a stored fact bank corrects attribution errors, and report that ParaMem outperforms token-memory baselines on most question types in real-world benchmarks.
Critical strengths
- The problem is interesting: the paper studies how to parameterize user memory while still allowing the generated memories to be inspected.
- The experimental comparison is fair, with consistent settings across methods.
- The analysis is comprehensive and shows the trade-off between storage and accuracy.
- The paper is clearly written and easy to follow.
Critical weaknesses
- The authors conduct experiments only on LoCoMo, which makes it difficult to assess the generalization ability of the proposed method.
- According to Table 4, ParaMem + grounding performs very close to No-adapter gen + grounding, suggesting that the contribution of the adapter is very limited.
- In Table 3, the differences among most results are small. Could the authors report results from multiple runs, or provide the mean and standard deviation, to allow a more reliable comparison?
- The evaluation metrics are relatively limited. The authors could add evaluation results based on LLM-as-a-judge.
Questions for authors
please refer to the weakness section
4 comments
In Table 3, the differences among most results are small. Could the authors report results from multiple runs, or provide the mean and standard deviation, to allow a more reliable comparison?
We thank the reviewer for this suggestion. Running Table 3 with multiple seeds showed that the submitted DPO variants were within run-to-run noise of SFT. We traced this to one design choice and revised it.
What changed. Only the source of the rejected memories. Everything else is unchanged: the DPO objective, the SFT initialization, β, the trainer, the training queries, and the three negative types. Preference learning is also a separate ablation, so none of the main results are affected.
Why. The submitted negatives were memories mined from the user's fact store. Before DPO starts, the SFT adapter already assigns these negatives a much lower likelihood than the correct memory. Most of the DPO update therefore lowers the probability of memories the adapter rarely samples, and has little effect on what it recalls at inference. This is consistent with prior findings that DPO works better when the rejected responses are sampled from the model being trained [3,4]. We now take negatives from the adapter itself: it recalls memories for paraphrased training queries, and recalls that ground to a wrong fact become the rejected memories, each assigned to one of the three types by the fact it grounds to. All three types use the same hyperparameter grid setup, with configurations selected by two-fold cross-user selection.
| System | Avg | C1 | C2 | C3 | C4 | C5 |
|---|---|---|---|---|---|---|
| SFT (no preference learning) | 0.348 ± 0.000 | 0.262 ± 0.011 | 0.334 ± 0.006 | 0.197 ± 0.026 | 0.378 ± 0.006 | 0.387 ± 0.008 |
| DPO + BM25-hard negatives | 0.350 ± 0.002 | 0.262 ± 0.008 | 0.328 ± 0.017 | 0.197 ± 0.011 | 0.381 ± 0.005 | 0.397 ± 0.004 |
| DPO + cross-fact negatives | 0.351 ± 0.004 | 0.276 ± 0.005 | 0.328 ± 0.012 | 0.211 ± 0.010 | 0.382 ± 0.009 | 0.386 ± 0.011 |
| DPO + different-subject negatives | 0.354 ± 0.004 | 0.271 ± 0.007 | 0.339 ± 0.001 | 0.210 ± 0.017 | 0.378 ± 0.005 | 0.404 ± 0.007 |
| Qwen2.5-3B, DeepSeek teacher; mean ± sd over 3 inference seeds. |
- All three negative types now improve over SFT. Different-subject negatives remain the best, consistent with the paper's claim.
- The gains are modest by design. DPO only refines which stored memory the adapter recalls. We keep the preference update small so that it does not overwrite the facts learned in SFT.
We will revise Table 3 with new results in the next version.
The evaluation metrics are relatively limited. The authors could add evaluation results based on LLM-as-a-judge.
We thank the reviewer for the thoughtful suggestion. We re-evaluate the paper's exact outputs with three additional measures:
- the Mem0 LLM-judge prompt [5], run with two judges different from the teacher, to avoid self-preference: Qwen3.5-397B (J-Q) and gpt-oss-120b (J-G);
- BLEU-1;
- SBERT similarity.
| Qwen2.5-3B | J-Q | J-G | BLEU-1 | SBERT |
|---|---|---|---|---|
| FullContext | 0.267 | 0.270 | 0.202 | 0.375 |
| Summarization | 0.440 | 0.426 | 0.287 | 0.515 |
| LangMem | 0.265 | 0.265 | 0.186 | 0.422 |
| Mem0 | 0.329 | 0.325 | 0.218 | 0.431 |
| ReadAgent | 0.348 | 0.344 | 0.249 | 0.451 |
| SimpleMem | 0.425 | 0.418 | 0.268 | 0.488 |
| A-Mem | 0.382 | 0.382 | 0.301 | 0.529 |
| Param A-Mem | 0.280 | 0.284 | 0.242 | 0.476 |
| ParaMem-direct | 0.367 | 0.351 | 0.250 | 0.485 |
| ParaMem-context | 0.441 | 0.440 | 0.269 | 0.510 |
| +ParamRecall | 0.453 | 0.446 | 0.289 | 0.527 |
| +ParamRecall+grounding | 0.593 | 0.582 | 0.387 | 0.622 |
| Llama-3.1-8B | J-Q | J-G | BLEU-1 | SBERT |
|---|---|---|---|---|
| FullContext | 0.343 | 0.345 | 0.259 | 0.455 |
| Summarization | 0.521 | 0.506 | 0.338 | 0.572 |
| LangMem | 0.232 | 0.229 | 0.205 | 0.304 |
| Mem0 | 0.337 | 0.335 | 0.246 | 0.454 |
| ReadAgent | 0.434 | 0.434 | 0.302 | 0.515 |
| SimpleMem | 0.409 | 0.401 | 0.278 | 0.481 |
| A-Mem | 0.447 | 0.438 | 0.315 | 0.542 |
| Param A-Mem | 0.280 | 0.279 | 0.212 | 0.432 |
| ParaMem-direct | 0.386 | 0.374 | 0.261 | 0.497 |
| ParaMem-context | 0.486 | 0.466 | 0.293 | 0.547 |
| +ParamRecall | 0.495 | 0.486 | 0.299 | 0.554 |
| +ParamRecall+grounding | 0.609 | 0.594 | 0.389 | 0.626 |
- The main conclusion does not depend on the metric. Full ParaMem is best on every metric and on both backbones, and the two independent judges agree closely.
- Parametric recall alone is competitive. Without any fact store, ParaMem+ParamRecall outperforms every token-memory baseline under both LLM judges on Qwen2.5-3B, and is second only to Summarization on Llama-3.1-8B under the LLM judge.
We will add these columns to Tables 1 and 4.
[1] Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." COLM 2025.
[2] Gao, Luyu, et al. "Precise zero-shot dense retrieval without relevance labels." ACL 2023.
[3] Tajwar, Fahim, et al. "Preference fine-tuning of llms should leverage suboptimal, on-policy data." ICML 2024.
[4] Xu, Shusheng, et al. "Is dpo superior to ppo for llm alignment? a comprehensive study." ICML 2024.
[5] Chhikara, Prateek, et al. "Mem0: Building production-ready ai agents with scalable long-term memory." arXiv preprint 2025.
Thank you for the response. After reading the results and clarifications, most of my concerns have been resolved. I will raise my score. Thanks to the authors' efforts in the rebuttal phase.
We thank the reviewer for the thoughtful feedback. We believe that our paper will be much stronger thanks to the reviewer's efforts.
We thank the reviewer for this suggestion and have added a second benchmark, PersonaMem (32k split) [1]. It contains 589 multiple-choice questions over 37 user histories and covers 7 question types, such as preference evolution, recommendation, and recall of shared facts.
The main findings transfer.
We will add these results as a new appendix section.
We thank the reviewer for the sharp observation. It exposed a flaw in how we set up Table 4: the comparison there is not fully controlled. Due to an oversight, the no-adapter control drew twice as many memory samples per question as ParaMem: kq=4 probes × ks=3 samples, vs. kq=2 probes × ks=3 samples. Once the sampling budget is matched, the adapter's contribution is clear and distinct.
Controlled comparison. We re-ran the control with exactly ParaMem's sampling budget: the same number of probes and samples and the same sampling settings. The only differences are the absence of the adapter and a HyDE-style prompt [2] asking the backbone to write a plausible memory.
(a) Table 4 setting. With the sample-matched control, ParaMem + grounding distinctly outperforms no-adapter gen + grounding on both backbones, where both use the same sampling budget (kq=2, ks=3).
(b) Matched fact budget. To isolate the adapter's contribution, every method receives exactly K facts, all retrieved by BM25 from the same fact store; the methods differ only in the BM25 query: the question (BM25), the untrained backbone's generated memories (no-adapter), or the adapter's recalled memories (ParaMem). For the latter two, the sampling setups are the same as before (kq=2, ks=3), and each generated memory is replaced by its top-1 BM25 fact; these facts come first, and BM25 (question) fills any remaining slots. F1 overall:
We will replace the Table 4 control with the sample-matched one, add the matched-budget table to the main text, and rewrite the discussion. We thank the reviewer again for prompting this correction.