acceptodds
Under review as a conference paper at ICLR 2027

Parameters as Agentic Memory: Internalizing Long-Horizon Memories for Efficient LLM Agents

Abstract

LLM agents need long-term memory, but most current systems keep memory outside the model and repeatedly append retrieved text to the prompt, which burdens both long-context reasoning and serving efficiency. We propose Parameters as Memory (PAM), a high-level methodology that treats durable agent knowledge as memory that can be internalized into parameter-efficient adapters. Inspired by neuroscience views of replay and consolidation, we instantiate PAM with Reflective Memory Optimization (REMO), a data rewriting and training pipeline that treats reflection over dialogue history as memory-oriented optimization. REMO reconstructs overlapping dialogue windows, rewrites them through intrinsic, cross-turn, network-structuring, and interaction-oriented views, and then trains LoRA adapters with supervised fine-tuning so the model better internalizes the rewritten memories. Across LoCoMo, LongMemEval, and MemoryAgentBench, PAM improves over the base model and non-rewriting parameter-memory baselines; it is especially effective for temporal reasoning, test-time learning, recommendation, summarization, and conflict resolution, while explicit retrieval remains stronger for verbatim lookup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Long-Term Memory, Parametric Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cbNLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Parameters as Memory (PAM), a methodology that internalizes stable long-term agent memories into parameter-efficient adapters instead of repeatedly retrieving and inserting historical text into the prompt. The authors instantiate PAM through Reflective Memory Optimization (REMO), which divides long conversations into overlapping windows, normalizes entities and temporal information, and rewrites each window using multiple complementary views, including direct and inverse factual expressions, cross-turn reasoning chains, question-answer representations, and structured entity or timeline summaries. The resulting corpus is used to train LoRA adapters while keeping the backbone model frozen. At inference time, the model can recall historical information from the adapter without receiving the complete conversation history. Experiments on LoCoMo, LongMemEval, and MemoryAgentBench demonstrate improvements over the base model and naive parameter-memory training, particularly for temporal reasoning, personalization, summarization, and conflict resolution. However, explicit retrieval remains stronger for exact lookup and rapidly changing information, suggesting that parameter memory should complement rather than replace external memory.

Critical strengths
  1. A practical alternative to external memory. PAM introduces a clear framework for storing stable, frequently reused memories in lightweight LoRA adapters, reducing the need to repeatedly retrieve and insert long histories into the prompt.
  2. Well-designed memory rewriting pipeline. REMO constructs multiple access paths to the same memory through forward and inverse expressions, temporal reasoning, cross-turn composition, QA encoding, and entity-level organization. The controlled ablations show that the gains cannot be explained solely by QA-format alignment.
  3. Comprehensive empirical and efficiency evaluation. The paper evaluates two backbone models across LoCoMo, LongMemEval, and MemoryAgentBench, includes component and LoRA-rank ablations, and reports practical serving metrics such as prompt length, latency, KV-cache usage, storage, and offline training cost.
Critical weaknesses

Limited methodological novelty and unclear gain attribution. PAM primarily combines established techniques—sliding-window preprocessing, LLM-based data augmentation, and standard LoRA supervised fine-tuning—rather than introducing a new learning algorithm. Moreover, REMO produces substantially more training data than the NaiveSFT baseline, but the paper does not provide a data-volume-matched comparison using simpler paraphrasing or generic QA augmentation. Therefore, it is difficult to determine whether the gains arise from the proposed reflective strategy taxonomy or simply from additional synthetic supervision.

Limited practicality for dynamic, large-scale memory. PAM requires generating training data and updating a dedicated adapter before new information becomes usable. The reported adapter occupies approximately 1.34 GB, and the system does not implement automatic routing between parametric and external memory, reliable deletion, or explicit unlearning of outdated facts. These limitations make the approach less suitable for rapidly changing, privacy-sensitive, or exact-match information; indeed, RAG substantially outperforms PAM on LongMemEval accuracy.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer cbNL [1/3]

We thank the reviewer for recognizing PAM's motivation, multi-view rewriting pipeline, and evaluation. To address gain attribution and practicality, we added matched-budget controls, continual-update experiments, and construction and query cost measurements.


Response to Question 1

Do the gains arise from REMO's reflective strategies or simply from additional synthetic supervision? Can the authors compare with simpler paraphrasing and generic QA augmentation under matched training-token budgets?

Response:

PAM contributes a supervision design and writing pipeline for long-term parametric memory. REMO organizes dispersed dialogue history into complementary views, including direct and inverse facts, cross-turn relations, temporal information, QA, and entity structures. Adapters internalize this information as reusable memory for subsequent generation, reducing repeated access to source history at query time.

1. Data construction under matched training budgets. We used Qwen2.5-7B-Instruct on the first three LoCoMo conversation histories, with three training conditions:

  • Naive SFT (raw dialogue): Train directly on the original session text, repeating it to reach the prescribed training-token budget.
  • Training on generated QA pairs: Use Qwen2.5-7B-Instruct to generate short QA pairs from facts explicitly stated in each source session, then train the adapter on these pairs.
  • REMO: Train the adapter on multi-view rewrites covering facts, inverse relations, cross-turn connections, temporal information, and QA.

All conditions use the same backbone, rank-128 LoRA, optimizer, learning-rate schedule, and random seed, with 600 updates and 1,228,800 non-padding input tokens. The three histories contain 152, 81, and 152 evaluation questions. The metric is lexical F1, averaged equally across histories.

Training dataLoCoMo Item 1LoCoMo Item 2LoCoMo Item 3Average F1
Naive SFT (raw dialogue)9.369.358.218.98
QA pairs generated from source sessions36.6238.9627.9534.51
REMO41.7149.9127.1739.59

Under the same training-token budget, REMO outperforms Naive SFT on all three histories, improving average F1 by 30.62 points. Converting sessions into QA pairs raises average F1 from 8.98 to 34.51; REMO reaches 39.59, a further 5.08-point improvement over QA training. These results show that data construction affects memory QA performance and that REMO's gains do not arise solely from more training tokens. Full settings are provided in Supplementary Experiments S1–S2.

2. Comparisons with other memory internalization pipelines. We added task adaptations of EntiGraph, Active Reading, and on-policy self-distillation (OPSD). EntiGraph generates relational descriptions and summaries around source entities and their combinations. Active Reading plans study questions and strategies, then generates paraphrases, QA, and other exercises. In OPSD, the student samples responses and learns from a frozen teacher that additionally reads the source session.

LoCoMo results with Qwen2.5-7B-Instruct. Metric: lexical F1.

MethodLoCoMo Item 1LoCoMo Item 2LoCoMo Item 3Average F1
PAM (REMO)39.1045.3334.3039.58
OPSD9.447.214.296.98
EntiGraph19.1011.0910.6713.62
Active Reading26.7025.0028.1426.61

For this comparison, REMO, EntiGraph, and Active Reading use newly generated corpora from Qwen2.5-7B-Instruct, with matched corpus token volumes, adapter configurations, and training budgets. These runs are independent of those in the preceding table. OPSD uses a separate distillation setup. REMO achieves the highest F1 on all three histories in this comparison, exceeding EntiGraph and Active Reading by 25.96 and 12.97 points on average. Configurations and other evaluation results are provided in Supplementary Experiments S3–S4.

These experiments evaluate REMO against raw-history training, direct data augmentation, and alternative internalization pipelines. They support improving parametric memory through supervision design and show how training corpora and internalization methods affect memory QA performance.

AuthorsOpenReview ↗

Response to Reviewer cbNL [2/3]

Response to Question 2

How practical is PAM for dynamic, large-scale, or privacy-sensitive memory, given data-generation and adapter-update costs, adapter storage, missing routing/deletion mechanisms, and the LongMemEval gap?

Response:

PAM internalizes reusable historical experience in parameters, exchanging offline writing for less historical context processing at query time. We evaluate continual updates, storage and query costs, and integration with external memory.

1. Continual updates and memory retention. PAM maintains an independent adapter for each user or memory unit, with a shared frozen backbone. New history is written by continuing to train the corresponding adapter.

We divided each of the first three LoCoMo histories chronologically into three stages and trained its adapter sequentially. Questions are grouped by when their supporting evidence first becomes available. The table reports F1 averaged equally across the three histories.

Adapter stateStage 1 questionsStage 2 questionsStage 3 questions
Base model7.755.515.47
After Stage 131.3316.0317.30
After Stage 1 → 222.1246.0418.02
After Stage 1 → 2 → 322.4529.8742.68

Writing Stage 2 and Stage 3 improves F1 on their corresponding questions by 30.00 and 24.66 points, respectively, showing that an existing adapter can continually absorb new information. Continued writing also affects earlier memories. We therefore compared replay and cumulative-history rebuilding on LoCoMo Item 1 under the same total training budget.

LoCoMo Item 1: F1 after the Stage 3 update.

Maintenance strategyStage 1 questionsStage 2 questionsStage 3 questions
Continued training without replay30.3730.2040.10
Continued training with 20% replay46.0535.3037.50
Adapter rebuilt from cumulative history48.3646.0238.37

Using 20% replay raises Stage 1 / 2 F1 to 46.05 / 35.30; cumulative-history rebuilding further raises these scores to 48.36 / 46.02. Under the fixed budget, replay allocates part of training to older data, shifting Stage 3 F1 from 40.10 to 37.50 and illustrating the trade-off between new learning and retention.

Continued training, replay, and cumulative-history rebuilding thus provide different maintenance options. The current implementation does not provide automatic routing or reliable fact-level deletion; information requiring precise deletion should remain in a managed external memory store.

AuthorsOpenReview ↗

Response to Reviewer cbNL [3/3]

2. Construction, storage, and query costs. We ran the complete 20-strategy REMO pipeline with Qwen2.5-7B-Instruct and trained rank-128 LoRA adapters on H200 GPUs. Construction times below are in minutes and include interrupted attempts.

DatasetData generationAdapter trainingTotal construction time
LoCoMo Item 174.8415.6690.50
LoCoMo Item 251.3117.6368.94
LoCoMo Item 3100.1916.96117.16

LoCoMo Item 1: storage and loading costs.

MethodStorage sizeLoading time (s)
PAM (REMO)1.292 GB2.385
BM25, including source text0.177 MB0.41

PAM loading measures attachment to an already resident backbone. At this adapter size, 1,000 users require approximately 1.292 TB of adapter storage. Storage and loading therefore remain part of the maintenance cost at scale.

For query measurements, we selected 128 fixed questions from Item 1 and alternated paired measurements over three rounds on the same shared H200 GPU, with batch size 1. Models and adapters remained resident; PAM used unmerged FP32 LoRA. Each call generated exactly 32 tokens, with prefix-KV reuse disabled. Values are medians. Generation time includes prefill and excludes retrieval, tokenization, loading, and network costs.

RAG input conditionPAM generation time (s)RAG generation time (s)PAM prompt KV (MiB)RAG prompt KV (MiB)
Actual BM25 top-5, 1,592.5 tokens0.9100.4631.0487.09
8,192 tokens0.9090.6621.04448.00
16,384 tokens0.9090.9541.04896.00
24,576 tokens0.9101.2971.041,344.00
32,736 tokens0.9101.6561.041,790.25

PAM receives only the question, with a median input length of 19 tokens. Fixed-length inputs concatenate the same user's source text in BM25 relevance order, cycling it when needed to measure the computational cost of long inputs. RAG generates faster with short contexts. Under this configuration, latencies are close around 16k input tokens, after which PAM's advantage grows with input length. At 32,736 tokens, PAM achieves a paired speedup of 1.82×.

Counting parameters and prompt KV together, the memory crossover occurs when RAG uses 22,528 more input tokens than PAM, offsetting PAM's additional LoRA parameters. At 32,736 tokens, PAM saves approximately 557 MiB across these two components.

These measurements quantify the parametric-memory trade-off: offline construction and adapter maintenance reduce the context, computation, and KV costs of repeatedly processing history in subsequent queries.

3. LongMemEval and complementarity with external memory. In the original LongMemEval experiment, PAM achieves 25.8% accuracy and RAG 52.5%. The benchmark requires recovering specific facts, temporal evidence, and cross-session details. RAG supplies source evidence at query time, while PAM recovers information from parameters. Explicit retrieval directly supports such precise access.

We evaluated their combination on LongMemEval-S using Qwen2.5-7B-Instruct. Each question's full candidate history was used to train an independent rank-32 LoRA through four rewriting views: factual restatements, temporal information, user attributes, and source-derived QA. RAG and RAG+PAM used identical BM25 top-5 sessions and prompts, differing only in whether the corresponding adapter was enabled. This supplementary experiment reports F1 separately from the original accuracy results.

MethodF1
RAG15.61
PAM13.03
RAG + PAM26.53

With the same retrieval budget, RAG+PAM improves F1 from 15.61 to 26.53, a gain of 10.92 points. The original LoCoMo experiments with LLaMA-3.1-8B also demonstrate this complementarity:

External memory systemAlone, F1With PAM, F1
A-Mem23.9734.63
Mem026.7039.20

PAM learns reusable historical experience, while external memory provides explicit access to specific evidence and recent information. These results support combining parametric memory and retrieval as complementary access mechanisms for different memory needs.

Reviewer_nxv6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Parameters as Memory (PAM), a framework that uses LoRA and supervised fine-tuning (SFT) to encode long-term user memories directly into model parameters, reducing reliance on external memory retrieval and historical context during inference. Its concrete implementation, REMO, divides dialogue histories into overlapping windows and constructs training data through factual rewriting, cross-turn synthesis, question–answer generation, and global structuring. Experiments on three memory benchmarks show improvements over fine-tuning on raw histories across various tasks, along with lower inference overhead. However, the method remains less effective than external retrieval methods on tasks requiring precise information retrieval.

Critical strengths
  1. The method explores an alternative to external memory systems such as A-Mem by internalizing long-term user memories into model parameters to improve the efficiency of memory agents.

  2. The experiments show strong results and include comprehensive ablation analyses. The method achieves gains across several memory benchmarks, particularly on tasks involving temporal reasoning, knowledge integration, and adaptation.

Critical weaknesses
  1. The end-to-end efficiency and cost require further evaluation. The method requires training user-specific adapters, which involves data construction and potentially periodic retraining as user interests and memories evolve. Compared with lightweight inference-time retrieval, how do the authors assess efficiency and cost across the entire pipeline?

  2. The handling of conflicting memories across users needs clarification. Different users may have distinct or conflicting long-term memories. Are their adapters trained and maintained independently? If an adapter is shared or sequentially updated across users, could standard SFT cause catastrophic forgetting of previously learned user memories?

  3. Comparisons with alternative memory internalization methods are missing. For example, how does the proposed SFT approach compare with on-policy self-distillation adapted to internalize user memories? Such a comparison would help establish the relative benefits of the proposed training approach.

Questions for authors

Please refer to my weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer nxv6 [1/2]

Thank you for raising lifecycle costs, cross-user memory conflicts, and alternative internalization methods. We added cost measurements, chronological update experiments, and method comparisons.


Response to Question 1

The end-to-end efficiency and cost require further evaluation. Compared with lightweight inference-time retrieval, how do the authors assess efficiency and cost across the entire pipeline?

Response:

PAM processes history offline and stores reusable memories in adapters, reducing historical context at query time. We assess writing and maintenance costs together with subsequent savings from shorter contexts.

1. Initial construction and storage. We ran the complete 20-strategy REMO pipeline on LoCoMo items 1–3, using Qwen2.5-7B-Instruct as both the local generator and backbone, and trained rank-128 LoRA adapters on H200 GPUs.

PAM (REMO) construction costs on LoCoMo. Times are in minutes.

DatasetGeneration timeTraining timeTotal construction time
LoCoMo item 174.8415.6690.50
LoCoMo item 251.3117.6368.94
LoCoMo item 3100.1916.96117.16

LoCoMo item 1 used 5.5k generation prompts, with 4M input tokens and 759k output tokens. Times include interrupted attempts.

LoCoMo item 1: memory storage and loading costs.

MethodStorage sizeLoading time (s)
PAM (REMO)1.292 GB2.385
BM250.177 MB0.41

PAM loading attaches an adapter to a resident backbone; BM25 loading restores the index and source text. Offline writing reduces repeated history processing, prefill, and KV costs at query time, as quantified below.

2. Query costs. We selected 128 fixed questions from LoCoMo item 1 and alternated paired measurements over three rounds on the same shared H200 GPU, with batch size 1. Models and adapters remained resident, and PAM used unmerged FP32 LoRA. Both methods used plain prompts, generated exactly 32 output tokens, and disabled prefix-KV reuse. Entries are medians. Generation time includes prefill and excludes retrieval, tokenization, loading, network, and queueing.

LoCoMo item 1: query costs with actual BM25 top-5 retrieval.

MethodInput tokensPrompt KV memory (MiB)Prefill time (ms)Generation time (s)
PAM (REMO), unmerged191.0432.850.910
RAG (BM25 top-5)1,592.587.0946.320.463

With short contexts, PAM uses less historical input, KV memory, and prefill time, while the current unmerged LoRA implementation incurs higher full-generation costs.

We extended the RAG reader input to 32,736 tokens, reserving space for 32 output tokens within the native context window. We concatenated the same user's source text in BM25 relevance order and cycled it when exhausted, isolating the computational cost of input length.

LoCoMo item 1: costs of generating 32 output tokens as RAG input length increases. PAM receives only the question, with median input length 19 tokens and prompt KV memory 1.04 MiB. The 15,616-token point was measured in a later refinement phase with timing drift relative to the initial measurements.

Input tokensPAM generation time (s)RAG generation time (s)RAG prompt KV memory (MiB)
8,1920.9090.662448.00
15,6161.0410.973854.00
16,3840.9090.954896.00
24,5760.9101.2971,344.00
32,7360.9101.6561,790.25

In this test, latencies are close around 16k input tokens, and PAM's query advantage grows as RAG input increases. At 16,384 tokens, PAM and RAG take 0.909 s and 0.954 s. At 32,736 tokens, PAM achieves a paired speedup of 1.82×, with prompt KV memory of 1.04 MiB versus RAG's 1,790.25 MiB. This captures a central benefit of parametric memory: learning from history offline reduces the computation and KV costs of repeatedly reading long histories during queries.

3. Why develop parametric memory alongside retrieval? PAM encodes reusable experience in parameters that condition generation. REMO organizes facts, temporal relations, and cross-turn connections into training views for reuse with less repeated history input. Retrieval accesses relevant evidence; parametric memory learns how experience shapes subsequent responses and decisions. The two mechanisms are complementary.

AuthorsOpenReview ↗

Response to Reviewer nxv6 [2/2]

Response to Question 2

Are different users' adapters trained and maintained independently? If an adapter is shared or sequentially updated across users, could standard SFT cause catastrophic forgetting?

Response:

PAM independently trains and maintains an adapter for each user or memory unit, while sharing a frozen backbone. Each user's memories are written to their own adapter, which is loaded for serving that user.

PAM updates a user's memories by further training the existing adapter. We tested continual updates on the first three LoCoMo histories separately. Within each history, we divided memory chronologically into item 1, item 2, and item 3 and wrote them sequentially to its adapter. For each item, we compute F1 per history and average equally across histories.

First three LoCoMo histories: average F1 after sequentially writing items 1, 2, and 3.

MethodItem 1 F1Item 2 F1Item 3 F1
Qwen2.5-7B7.755.515.47
PAM after item 131.3316.0317.30
PAM after item 1 → item 222.1246.0418.02
PAM after item 1 → item 2 → item 322.4529.8742.68

Writing items 2 and 3 improves F1 on their corresponding questions by 30.00 / 24.66 points. After applying common overlap exclusions and removing test QA pairs known to occur in static templates, the gains remain 27.95 / 24.30 points. After the item 3 update, F1 on all three items exceeds the base model, showing that continued training incorporates new information while retaining useful historical memories.

Continued writing also affects earlier memories. To improve retention, we compared three maintenance strategies on the first LoCoMo conversation history under the same training budget.

First LoCoMo history: F1 by item after the item 3 update.

MethodItem 1 F1Item 2 F1Item 3 F1
PAM without replay30.3730.2040.10
PAM with 20% replay46.0535.3037.50
PAM trained on cumulative history48.3646.0238.37

Adding 20% replay raises item 1 / item 2 F1 from 30.37 / 30.20 to 46.05 / 35.30; rebuilding the adapter from cumulative history achieves 48.36 / 46.02. Under the fixed training budget, replay allocates part of training to older data, with item 3 F1 changing from 40.10 to 37.50.

PAM supports incremental writing into an existing adapter, and replay and cumulative-history rebuilding improve retention. Separate adapters manage users' memories; continued training and replay maintain each user's evolving history.


Response to Question 3

How does the proposed SFT approach compare with on-policy self-distillation adapted to internalize user memories?

Response:

We evaluated OPSD and added adaptations of EntiGraph [1] and Active Reading [2]. EntiGraph expands training corpora through diverse relational descriptions and summaries of source entities, pairs, and triples. Active Reading plans study questions and strategies from source documents, then generates paraphrases, QA, and other exercises for factual internalization.

In our OPSD implementation, the student samples answers to training questions. A frozen backbone teacher additionally reads the corresponding source session. We minimize forward KL from teacher to student on the same student-sampled answer prefixes.

LoCoMo results with Qwen2.5-7B-Instruct. Items 1, 2, and 3 contain 152, 81, and 152 evaluation questions. Average F1 weights the three items equally.

MethodLoCoMo item 1 F1LoCoMo item 2 F1LoCoMo item 3 F1Average F1
PAM (REMO)39.1045.3334.3039.58
OPSD9.447.214.296.98
EntiGraph19.1011.0910.6713.62
Active Reading26.7025.0028.1426.61

PAM (REMO), EntiGraph, and Active Reading use the same backbone, adapter configuration, and training budget, with matched training-data volumes. EntiGraph retains generated summaries and covers entity combinations; Active Reading executes every complete strategy in the accepted plans. OPSD uses a separate experimental setup. All methods are evaluated on memory QA using lexical F1; configurations and other evaluation results are in Supplementary Experiments S3–S4.

PAM achieves the highest F1 on all three LoCoMo histories, averaging 39.58 and exceeding EntiGraph and Active Reading by 25.96 / 12.97 points. REMO builds multi-view training corpora through factual paraphrasing, temporal relations, and cross-turn integration, achieving better memory QA under the same training budget and demonstrating the strength of its rewriting and internalization pipeline.


References

[1] Yang et al. Synthetic continued pretraining. ICLR 2025.

[2] Lin et al. Learning Facts at Scale with Active Reading. ICLR 2026.

AuthorsOpenReview ↗

Author Follow-up for Reviewer nxv6

Dear Reviewer nxv6,

Thank you again for your time and effort in reviewing our submission.

As the discussion period will end in about two days, we would like to kindly remind you that we have posted our responses to all of your comments. If you have any further questions or concerns, we would be happy to discuss them.

We sincerely appreciate your valuable feedback and thank you again for your time and consideration.

Reviewer_B8dHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Parameters as Memory (PAM), a framework that internalizes long-term agent memories into parameter-efficient LoRA adapters instead of relying on external retrieval. The authors introduce Reflective Memory Optimization (REMO), which transforms dialogue histories into multiple training perspectives, including factual recall, temporal reasoning, and cross-turn relationships. Experiments on LoCoMo, LongMemEval, and MemoryAgentBench demonstrate improvements over naive fine-tuning and several memory baselines.

Critical strengths
  • The paper explores an interesting direction for reducing the inference overhead of external memory retrieval by internalizing historical information into model parameters.
  • The proposed multi-perspective memory rewriting strategy is well motivated and improves memory recall compared with naive supervised fine-tuning.
  • The experiments cover multiple benchmarks and memory-intensive tasks, with ablation studies examining different rewriting strategies and their contributions.
Critical weaknesses
  • The methodological novelty is relatively limited, as the approach primarily combines existing data augmentation strategies with standard LoRA fine-tuning.
  • The improvements may partly result from substantially more augmented training data, but the paper lacks controlled comparisons with equal training-token budgets.
  • The method significantly underperforms RAG on LongMemEval (25.8 vs. 52.5 accuracy), raising concerns about whether parameterized memory can reliably replace external retrieval.
  • There are unexplained discrepancies between experimental tables, particularly the NaiveSFT and full PAM results in Tables 1 and 3, which make the ablation results difficult to interpret.
  • The claimed serving efficiency benefits are not sufficiently demonstrated in the main text, especially considering the training and maintenance costs of user-specific adapters.
Questions for authors
  • Can the authors provide comparisons against naive SFT with matched training-token budgets to isolate the benefits of the proposed rewriting strategies?
  • Could the authors clarify the substantial discrepancies between the results reported in Tables 1 and 3?
  • How does PAM handle frequent memory updates, conflicting information, or forgetting outdated memories without retraining adapters?
  • What are the end-to-end computational and storage costs of maintaining personalized LoRA adapters compared with conventional retrieval-based memory systems?
5 comments
AuthorsOpenReview ↗

Response to Reviewer B8dH [1/3]

We thank the reviewer for the constructive comments. We address each question below with additional experiments and clarifications.


Response to Question 1

Can the authors provide comparisons against naive SFT with matched training-token budgets to isolate the benefits of the proposed rewriting strategies?

Response:

Thank you for this suggestion. Matching training-token budgets controls for performance differences caused by greater training exposure.

We evaluated Qwen2.5-7B-Instruct on the first three LoCoMo item histories. For each history, all methods used approximately 1.2 million training tokens and the same evaluation questions. We also included Simple Rewriting, which converts the same conversations into ordinary narrative paraphrases, as a control for REMO's hierarchical rewriting. The metric is F1.

TaskNaive SFTSimple RewritingREMO
LoCoMo Item 19.3616.2941.71
LoCoMo Item 29.3516.1449.91
LoCoMo Item 38.2118.7527.17

With the training-token budget held fixed, REMO outperforms both controls on all three histories, improving over Naive SFT by 32.35, 40.56, and 18.96 F1 points, respectively. These results show that REMO organizes the same history into more effective supervision, improving information integration and memory QA through hierarchical rewriting.


Response to Question 2

Could the authors clarify the substantial discrepancies between the results reported in Tables 1 and 3?

Response:

Table 1 compares PAM with other baselines using the average over all ten LoCoMo conversation histories.

Table 3 is an ablation of the contribution of QA-format rewriting to PAM. It uses one representative LoCoMo history to examine whether the gains come solely from QA-formatted training data. This ablation evaluates PAM's benefits beyond QA formatting.

The tables therefore serve different purposes and cover different evaluation scopes. We will clarify these distinctions in the captions and main text.


Response to Question 3

How does PAM handle frequent memory updates, conflicting information, or forgetting outdated memories without retraining adapters?

Response:

PAM maintains an independent LoRA adapter for each user or memory unit, with a shared frozen backbone. New history is written by continuing to train that user's existing adapter.

We conducted continual-update experiments separately on LoCoMo Items 1–3. Each history was divided chronologically into three stages, and the same adapter was trained sequentially on these stages. Questions are grouped by when their supporting evidence becomes available, and F1 is averaged equally across the three histories.

LoCoMo Items 1–3: mean F1 during continual updating.

MethodStage 1 F1Stage 2 F1Stage 3 F1
Qwen2.5-7B7.755.515.47
PAM after Stage 131.3316.0317.30
PAM after Stage 1 → 222.1246.0418.02
PAM after Stage 1 → 2 → 322.4529.8742.68

Updating the adapter with Stage 2 and Stage 3 history raises F1 on the corresponding question groups by 30.00 and 24.66 points, respectively. With common exclusions for overlapping evaluation questions and known test QA examples in the static templates, these gains remain 27.95 and 24.30 points.

These results demonstrate that PAM continually absorbs new information within the same adapter. After all three updates, PAM outperforms the base model on all three question groups. Continual writing also introduces interference with earlier memories, so we evaluated retention strategies under the same training budget on LoCoMo Item 1.

LoCoMo Item 1: F1 after the third-stage update.

MethodStage 1 F1Stage 2 F1Stage 3 F1
PAM without replay30.3730.2040.10
PAM with 20% replay46.0535.3037.50
PAM trained on cumulative history48.3646.0238.37

Using 20% replay raises F1 on the first two stages from 30.37 and 30.20 to 46.05 and 35.30. Rebuilding on cumulative history further raises these scores to 48.36 and 46.02. Under a fixed training budget, replay allocates some training exposure to older data, shifting Stage 3 F1 from 40.10 to 37.50 and illustrating the trade-off between new learning and retention.

PAM supports incremental writing with concrete strategies for preserving earlier knowledge. Continued training, replay, and cumulative rebuilding provide maintenance options for different memory-update requirements.

AuthorsOpenReview ↗

Response to Reviewer B8dH [2/3]

Response to Question 4

What are the end-to-end computational and storage costs of maintaining personalized LoRA adapters compared with conventional retrieval-based memory systems?

Response:

PAM internalizes historical information in model parameters, reducing the history that must be reintroduced into the LLM context at each query. This reduces repeated long-context processing, including the quadratic prompt-length component of dense self-attention during prefill, and lowers context-budget and per-query KV-cache requirements for long-horizon tasks.

We measured offline writing, storage and loading, query latency, and KV memory.

1. Offline construction, storage, and loading. On LoCoMo Items 1–3, we used Qwen2.5-7B-Instruct for the full 20-strategy REMO writing pipeline and trained rank-128 LoRA adapters on H200 GPUs. The table reports cumulative construction times in minutes, including interrupted attempts.

DatasetGeneration timeTraining timeTotal construction time
LoCoMo Item 174.8415.6690.50
LoCoMo Item 251.3117.6368.94
LoCoMo Item 3100.1916.96117.16

For Item 1, the PAM adapter occupies 1.292 GB, compared with 0.177 MB for the BM25 index including source text. Loading takes 2.385 s and 0.41 s, respectively; PAM loading measures attachment to an already resident backbone. PAM incurs higher offline construction and parameter-maintenance costs in exchange for reusing history across subsequent queries.

AuthorsOpenReview ↗

Response to Reviewer B8dH [3/3]

2. Query latency and KV memory. We selected 128 fixed questions from LoCoMo Item 1 and performed three rounds of alternating paired measurements on one shared H200 GPU, with batch size 1. Models and adapters were resident; PAM used an unmerged FP32 LoRA. Each call generated exactly 32 output tokens, with prefix-KV reuse disabled. Values are medians, and generation time includes prefill. The timers cover model computation; retrieval, tokenization, loading, and network costs are excluded.

Query costs with actual BM25 top-5 retrieval.

MethodInput tokensPrompt KV memory (MiB)Prefill time (ms)Generation time (s)
PAM (REMO), unmerged191.0432.850.910
RAG (BM25 top-5)1,592.587.0946.320.463

For short contexts, BM25-based generation is faster, while PAM reduces the median input to 19 tokens and prompt KV to 1.04 MiB. We then increased the RAG reader's input length to measure the cost of processing longer histories.

Long-input stress test: both methods generate exactly 32 output tokens. RAG inputs concatenate the same user's source text in BM25 relevance order, cycling the text when needed to reach the target length. PAM receives only the question. The points below come from the same interleaved grid-measurement phase.

RAG input tokensPAM generation time (s)RAG generation time (s)RAG prompt KV memory (MiB)
8,1920.9090.662448.00
16,3840.9090.954896.00
24,5760.9101.2971,344.00
32,7360.9101.6561,790.25

Under this configuration, query latency crosses over around 16k input tokens, and PAM's advantage grows with longer RAG inputs. At 32,736 tokens, PAM achieves a 1.82× paired speedup and reduces prompt KV from 1,790.25 MiB to 1.04 MiB. Across this grid, PAM's generation time stays near 0.91 s as the RAG input grows.

PAM's additional LoRA parameters occupy 1,291,845,632 bytes, and KV memory uses 57,344 bytes per input token. Counting parameters and prompt KV together, the memory crossover occurs when RAG uses 22,528 more input tokens than PAM. At 32,736 tokens, PAM saves approximately 557 MiB across these two components, including its additional LoRA parameters.

The core benefit of parametric memory is to learn reusable historical knowledge offline and apply it through the adapter during future generation. Retrieval-based token memory locates relevant text and supplies it as context at each query; parametric memory directly reuses learned experience, reducing history replay and context usage. For long-horizon interactions, this keeps inputs short and response costs more stable as history grows. These benefits motivate offline writing and adapter maintenance.

PAM can be less accurate than RAG when queries require precise access to individual facts. PAM can also be combined with RAG, reducing reliance on long history replay while retaining explicit retrieval for source-specific information. The paper's integrations with external memory systems demonstrate this complementarity:

LoCoMo, LLaMA-3.1-8BRAG, F1RAG With PAM, F1
A-Mem23.9734.63
Mem026.7039.20

Why does PAM score below RAG on LongMemEval?

On LongMemEval, PAM achieves 25.8% accuracy and RAG 52.5%. This benchmark requires precise recovery of specific facts, temporal evidence, and details across sessions. RAG provides source passages that the model can read directly; PAM encodes history in parameters and recovers these details during generation. Retaining an explicit retrieval channel serves precise memory queries while complementing PAM's learned memory.

To evaluate the combination, we conducted a simplified reproduction with Qwen2.5-7B-Instruct on 30 fixed LongMemEval-S questions, five from each of six types. Each question's full candidate history was used to train an independent rank-32 LoRA. The four rewriting views were factual restatements, temporal information, user attributes, and source-derived QA; test questions and answers were not provided to the generator. RAG and RAG+PAM used identical BM25 top-5 sessions, prompts, and a 64-token output limit; the difference was whether the corresponding adapter was enabled.

LongMemEval-S, 30 questions, Qwen2.5-7B-Instruct. Metric: F1.

MethodF1
RAG15.61
PAM13.03
RAG + PAM26.53

With the same retrieval budget, RAG+PAM improves F1 from 15.61 to 26.53, a gain of 10.92 points. Parametric memory provides additional value for retrieval-augmented answering, combining learned experience reuse with explicit access to facts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the reviewers for their constructive feedback. We summarize the main concerns and our responses below.

  1. Contribution and gain attribution (B8dH, cbNL). Our contribution centers on REMO’s memory-oriented supervision and consolidation pipeline. On three LoCoMo histories, controlling training exposure at 1.23M tokens yields 39.59 average F1, versus 8.98 for raw-history SFT and 34.51 for source-derived QA; REMO also outperforms simple paraphrasing. In separate matched-budget comparisons, REMO achieves 39.58 F1, versus 13.62/26.61 for adapted EntiGraph/Active Reading. We additionally evaluated OPSD under a separate distillation setup. These controls support the value of REMO’s supervision design beyond increased training exposure.

  2. Efficiency and scalability (all reviewers). We added measurements of construction, storage, loading, latency, and KV memory. The tested configuration requires 69–117 minutes for initial construction and 1.292 GB per adapter. Short-context RAG generates faster. In the controlled long-input stress test at 32,736 tokens, PAM achieves 1.82× model-compute speedup and reduces prompt KV from 1,790.25 to 1.04 MiB; timings exclude retrieval, tokenization, loading, and networking. These results quantify the trade-off between offline maintenance and repeated context processing. Large-scale adapter storage and updating remain limitations, including those raised in nxv6’s follow-up.

  3. Memory updates and retention (all reviewers). Users have independent adapters sharing a frozen backbone. New memories require continued training of the corresponding adapter. Chronological-update experiments demonstrate new-memory acquisition, while replay improves retention: on LoCoMo Item 1, 20% replay raises earliest-stage F1 from 30.37 to 46.05. Reliable fact-level deletion and automatic memory routing remain open.

  4. Relationship to retrieval (B8dH, cbNL). We acknowledge RAG’s stronger original LongMemEval accuracy. PAM targets reusable historical knowledge, with retrieval providing precise evidence access. On a 30-question LongMemEval-S subset, enabling PAM with identical retrieval and prompts improves F1 from 15.61 to 26.53, supporting their complementarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.