acceptodds
Under review as a conference paper at ICLR 2027

RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents

27 views

Abstract

Long-running LLM agents require memory that persists and evolves across sessions. Text-based memory retrieves and reconstructs past interactions at every query, making long-horizon performance increasingly dependent on retrieval quality and contextual reasoning as histories grow. Parametric memory encodes experience directly into model computation, but existing approaches provide limited support for cross-session memory evolution. Their coupling to a specific backbone further restricts memory reuse after model replacement. We introduce RPMem, a two-stage architecture that compiles each session into a model-independent latent memory through forward computation and selectively integrates it with retained memory via a task-trained recurrent gate. The consolidated memory is then mapped to backbone-specific low-rank adaptation (LoRA) parameters, allowing the encoding capability to transfer when the backbone is replaced. Evaluation across three long-term memory benchmarks and five diverse backbones demonstrates broad generalization with near-constant update cost and memory footprint. With Qwen3-8B on PERMA, RPMem reaches 85.52%, outperforming the strongest parametric and text-based baselines by 5.32 and 12.98 percentage points, respectively. Ablations validate the complementary roles of session compilation and cross-session consolidation, while dynamics analyses reveal that the gate acquires task-specific memory integration strategies. These results establish RPMem as a lifecycle-independent parametric memory framework that maintains evolving cross-session memory that remains reusable across backbone replacements. Our implementation is available in an anonymous repository at [code](https://anonymous.4open.science/r/rpmem-submission-B176).

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Parametric Memory, Long-Term Memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_oQ9eOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes RPMem, a parametric memory framework for LLM agents. It first trains a session encoder (consisting of a frozen context encoder and a trainable resampler) and a decoder to compress individual interaction sessions into latent representations that are mapped to LoRA weights. A second training stage learns a recurrent aggregation mechanism that combines memories across sessions into a fixed-size representation. The accumulated memory can also be reused across different LLM backbones by adapting only the decoder. Experiments on three long-term memory benchmarks show improvements over existing text-based and parametric memory methods, with approximately constant memory size and update cost.

Critical strengths
  1. Interesting combination of existing techniques: The paper combines context-to-LoRA compilation, previously explored by Doc-to-LoRA [1] and SHINE [2], with incremental latent memory updates, previously explored in approaches such as MemoryLLM [3] and M+ [4]. RPMem introduces a learned recurrent consolidation mechanism and separates accumulated memory from backbone-specific LoRA decoding, enabling memory reuse across model replacements.

  2. Strong empirical results and efficiency: RPMem achieves competitive or superior performance against existing text-based and parametric memory methods across three benchmarks. The ablations show substantial benefits from learned consolidation, while the approximately constant memory footprint and update cost suggest practical advantages for long-running agents.

[1] Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, and Robert Tjarko Lange. Doc-to-LoRA: Learning to instantly internalize contexts. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=iW1oBBO72S.

[2] Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelberg, Haggai Maron, and Muhan Zhang. SHINE: A scalable in-context hypernetwork for mapping context to LoRA in a single pass. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026c. URL https://openreview.net/ forum?id=ZMexYcAibv.

[3] Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, and Julian McAuley. MEMORYLLM: Towards selfupdatable large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 50453–50466. PMLR, 2024c. URL https://proceedings.mlr.press/v235/wang24s.html.

[4] Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, and Zexue He. M+: Extending MemoryLLM with scalable long-term ´ memory. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 63308–63323. PMLR, 2025a. URL https://proceedings.mlr.press/v267/wang25au.html.

Critical weaknesses

I find the paper technically strong and the results promising. While the following limitations do not change my overall positive assessment, addressing them would help strengthen the novelty claims and practical motivation.

  1. Incomplete comparisons with closely related methods. The paper does not experimentally compare against Doc-to-LoRA [1], SHINE [2], or several existing latent-memory approaches (e.g., M+ [4]). This is particularly concerning because context-to-LoRA compilation (e.g., Doc-to-LoRA [1] does that) and recurrent latent memory updates (e.g., M+ [4] maintains a fixed-size pool of latent memory tokens within the transformer) have already been explored independently in other works. A comparison with Doc-to-LoRA [1] or SHINE [2] extended with recurrent aggregation would help establish whether RPMem's architecture provides meaningful improvements beyond combining existing techniques. Ideally, also compare recurrent consolidation with M+ [4], but I understand that it might be infeasible during the rebuttal as it would require significant architectural changes. These comparisons would help better isolate the contribution of RPMem's particular architecture beyond combining existing techniques.

  2. Limited practical benefits of lifecycle independence. Although RPMem preserves the accumulated latent memory across backbone replacements, adapting to each new backbone still requires training a target-specific decoder on 664K compilation sessions (10,377 optimizer updates). The paper demonstrates that reusing the learned encoder outperforms training a new compiler from scratch under the same target-side budget, but does not establish the total computational advantage of this approach relative to lightweight text-based memory, which requires no backbone-specific memory retraining. This leaves the practical significance of the claimed lifecycle independence only partially established.

Questions for authors
  1. Missing closely related baselines: Why are Doc-to-LoRA [1], SHINE [2] not included in the experimental comparison? In particular, could the authors evaluate context-to-LoRA methods extended with recurrent aggregation to better isolate the novelty of RPMem?

  2. Unexpected Full Context degradation: Why does Qwen3.5-35B-A3B perform substantially worse with Full Context than with No Context in Figure 2? Could the authors verify these results and explain the degradation?

  3. Functional validation of latent memory: The memory-dynamics analysis shows correlations between gate coefficients and semantic event types, but does not directly establish which facts are preserved in the latent state. Could the authors conduct controlled experiments with synthetic facts and targeted memory interventions to demonstrate causal retention and retrieval?

  4. Generalization of recurrent consolidation: The consolidation gate is trained using task-specific supervision. How well does a gate trained on one benchmark or application generalize to another? Could the authors evaluate cross-task transfer without retraining the gate? This would help establish whether the learned memory-update mechanism is general-purpose or strongly dependent on the downstream task.

  5. The central lifecycle claim also involves continuing to update an existing memory after changing backbones. Do the authors explicitly evaluate the sequence: sessions on LLM A -> switch to B -> new sessions on LLM B -> check recall from memory? If your experiments only evaluate decoding an existing memory on a different backbone, that tests portability but not necessarily the full lifecycle scenario.

  6. Frozen encoder: The context encoder appears to be frozen while the resampler is trained. Could the authors clarify this distinction earlier in the paper and explain the motivation for freezing the encoder?

7 comments
AuthorsOpenReview ↗

Response to Reviewer oQ9e (1/3)

We thank the reviewer for the thoughtful evaluation and constructive suggestions. We address the related-method comparisons and practical value of lifecycle independence first, followed by the questions about evaluation and memory behavior.

W1 & Q1: Related Methods and Recurrent Aggregation

Thank you for suggesting additional comparisons with related methods to clarify RPMem's contribution beyond combining context-to-LoRA compilation with recurrent memory updates. We address this through the existing consolidation ablations and a controlled comparison with SHINE. Our consolidation ablations (Section 3.3) partly isolate this question: Latest Session compiles only the most recent session into LoRA parameters and achieves 53.56% on the four core PERMA settings, versus 85.52% for RPMem. Latent averaging and LoRA factor averaging test fixed cross-session aggregation rules using the same compiler.

We further evaluated the released SHINE-ift_mqa checkpoint on Qwen3-8B across all seven PERMA variants. SHINE w/o Gate uses fixed averaging of compiled session representations. SHINE w/ Gate freezes SHINE's compiler and decoder and adds the same recurrent consolidation gate architecture as RPMem. RPMem w/ Gate uses our fixed-reference forward-KL compiler. Both gates use the same ten-fold leave-one-user-out splits, answer supervision, and five training epochs.

Table 1. Comparison with SHINE on PERMA (accuracy, %).

MethodCore Avg.7-Variant Avg.
SHINE w/o Gate28.6429.26
SHINE w/ Gate84.2585.01
RPMem w/ Gate85.5286.53

Adding learned consolidation raises SHINE's seven-variant average from 29.26% to 85.01% (+55.75 percentage points, pp). Under matched gate architecture and downstream training, RPMem improves further by 1.52 pp on the seven-variant average and 1.27 pp on the core average.

Doc-to-LoRA and SHINE learn forward context-to-adapter mappings; M+ maintains updatable latent memory within the serving backbone. RPMem instead preserves a fixed-size recurrent state independently of the backbone and decodes it into model-specific LoRA parameters, allowing accumulated memory to continue across backbone replacements.

W2: Practical Value of Lifecycle Independence

Thank you for highlighting the need to quantify the practical computational cost of lifecycle independence, including decoder adaptation relative to text-based memory. We have added an amortized deployment-cost analysis covering decoder adaptation, online memory writing, and querying across different backbone replacement frequencies.

A new Qwen3.5-9B decoder costs 30.92 hours on eight GPUs (247.36 GPU-hours) to adapt, using 664K compilation sessions and 10,377 optimizer updates. It is trained once per target backbone and shared across users; existing user memory states, the session encoder, and the recurrent gate are retained. We compare with single-pass LightMem on PrefEval 300-turn, using measured query latencies of 146.24 ms (RPMem) and 326.88 ms (LightMem). RPMem takes 0.043 s per session write in its streaming profile; LightMem takes 177.63 s to ingest a 301-session history, averaging 0.590 s/session over the ingestion process.

For a one-year scenario with 100K users, 10 queries and one session write per user per day, we count adaptation for every new backbone and convert online timings to single-GPU-equivalent hours.

Table 2. Annual adaptation, writing, and querying cost (GPU-equivalent hours).

Backbone replacementRPMem totalLightMem totalReduction (%)
Annually15,510.4439,123.9460.36
Quarterly16,252.5239,123.9458.46
Monthly18,231.4039,123.9453.40
Weekly28,125.8039,123.9428.11

Even after including decoder adaptation, RPMem reduces the estimated annual computation by 53.40% with monthly and 28.11% with weekly backbone replacements.

AuthorsOpenReview ↗

Response to Reviewer oQ9e (2/3)

Q2: Full Context Degradation on Qwen3.5-35B-A3B

Thank you for highlighting the unexpected performance degradation of Full Context relative to No Context on Qwen3.5-35B-A3B. We have checked the reported results and added a Relevant Context evaluation to examine the effect of unrelated historical information.

Table 3. Qwen3.5-35B-A3B on PERMA (accuracy, %).

Context conditionAccuracy
No Context79.48
Full Context64.86
Relevant Context83.78
RPMem94.21

No Context, Full Context, and RPMem are the original results (Table E.11); Relevant Context is newly evaluated.

Full Context trails No Context by 14.62 percentage points (pp). Providing relevant history instead yields 83.78%, 4.30 pp above No Context. This pattern is consistent with interference from irrelevant history making it harder to use relevant information in the full context.

Q3: Functional Validation of Latent Memory

Thank you for suggesting a functional evaluation of whether the gate dynamics in Section 4 correspond to the retention and retrieval of specific information. We have added a controlled preference-intervention case study to examine how changing one memory input affects downstream predictions after subsequent memory updates.

Using Qwen3-8B with a trained PrefEval compiler and recurrent gate, we kept all parameters frozen and constructed two histories differing only in one target preference:

  • History A: Python learning resources must be accessible to visually impaired users; visually dependent materials should be avoided.
  • History B: The user dislikes audio/video learning resources and prefers written explanations and textbooks.

Both histories also contain the same independent preference for manual-transmission rental cars and the same 10 subsequent dialogue turns (20 messages). Each history is compiled and consolidated independently from the same initial state. We then ask two separate multiple-choice questions: “What are the best resources to learn Python programming?” (target) and “Can you recommend a reliable car rental model in Miami?” (control). The reader receives the questions and options, but no history text.

Table 4. Answers after 10 intervening dialogue turns.

HistoryPython resources (target question)Car rental (control question)
AAudiobooks for visually impaired learnersManual-transmission Volkswagen Golf
BPrinted Python booksManual-transmission Volkswagen Golf

Replacing the learning preference changes the target answer from audiobooks to printed books, while the independent car preference yields the same answer. The pattern holds for both the original and reversed option order. This controlled input intervention connects specific retained information to downstream predictions, complementing the gate-coefficient analysis in Section 4.

Q4: Cross-Task Transfer of the Consolidation Gate

Thank you for raising the question of whether a consolidation gate trained with downstream supervision can generalize to an unseen benchmark without target-task retraining. We examine this through zero-shot transfer of PERMA-trained consolidation gates to PersonaMem-v2. We transferred 70 PERMA-trained gates (seven variants × ten leave-one-user-out folds) to PersonaMem-v2 using RPMem with Qwen3.5-9B. The compiler, decoder, and gates were frozen; each gate was evaluated on all 5,000 target questions. We averaged accuracy across source folds and then equally across the seven source variants.

Table 5. Consolidation transfer to PersonaMem-v2 (accuracy, %).

ConsolidationTraining sourceAccuracy
Latent averagingNone35.25
Recurrent gatePERMA38.14
Recurrent gatePersonaMem-v245.73

Without PersonaMem-v2 gate training, the transferred gate reaches 38.14%, exceeding fixed latent averaging (35.25%) by 2.89 pp. Target-task training further improves accuracy to 45.73% (+7.59 pp), showing the additional benefit of adapting consolidation to the application's memory requirements.

AuthorsOpenReview ↗

Response to Reviewer oQ9e (3/3)

Q5: Continued Memory Updates after Backbone Replacement

Thank you for highlighting the distinction between preserving an existing memory state and continuing to update that memory after backbone replacement. We have extended the evaluation to test continued memory writing and querying while retaining the accumulated memory and source-trained consolidation gate.

We evaluated this sequence on PERMA Noisy SD, using ten leave-one-user-out folds and 695 questions with multi-session histories. We tested both Qwen3-4B → Qwen3-8B and Qwen3-8B → Qwen3-4B. For each history, we processed the first half of the sessions using backbone A and its source-trained gate (G_A). At the switch, we preserved the accumulated memory state, session encoder, and gate (G_A), replacing only the serving backbone and its adapted LoRA decoder. We then processed the remaining sessions and evaluated the final memory with backbone B.

We compare the retained source gate with a reference that uses the target backbone's trained gate (G_B) throughout the history.

Table 6. Continued memory updates across backbone replacement on PERMA Noisy SD (accuracy, %).

Backbone transitionRetained source gate (G_A)Target-trained gate (G_B)
Qwen3-4B → Qwen3-8B85.3791.37
Qwen3-8B → Qwen3-4B76.0286.00

With the original gate retained, RPMem achieves 85.37% after switching from 4B to 8B and 76.02% in the reverse direction, without retraining the gate or replaying previous sessions. The target-trained gates achieve 91.37% and 86.00%, respectively, showing the additional benefit of target-specific consolidation training.

Thus, adapting only the model-specific decoder supports continued memory updates and querying while preserving accumulated memory and the source gate, consistent with lifecycle independence.

Q6: Frozen Context Encoder and Trainable Resampler

Thank you for pointing out the need to distinguish the training roles of the frozen context encoder and trainable resampler, and to explain the motivation for this design. We have clarified these distinctions in Section 2.2. The session encoder comprises a pretrained, frozen context encoder that extracts token-level features and a trainable Perceiver-based resampler that compresses those features into a fixed-shape latent representation. In Single-Session Memory Compilation, the resampler and model-specific LoRA decoder are trained jointly through context-conditioned distribution matching, while the context encoder and answering backbone are frozen. Cross-Session Memory Consolidation then trains only the recurrent gate, with the compiler frozen.

Freezing the context encoder maintains a stable feature space as serving backbones change. The resampler learns a shared representation for session memories, while each backbone-specific decoder maps that representation to its own LoRA parameters. This allows the same encoder and accumulated memory to be retained when a new backbone is introduced.

We have incorporated the additional experiments and analyses into the revision and clarified the motivation for efficient cross-session memory access and the encoder–resampler training roles in the main text.

Reviewer_WWZQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RPMem (Recurrent Parametric Memory), a two-stage architecture designed to provide lifecycle-independent parametric memory for long-running LLM agents. The approach separates persistent memory from the serving backbone through: (1) a single-stage compilation module that encodes individual sessions into model-independent latent representations using a frozen context encoder and trainable Perceiver-based resampler, then decodes these into backbone-specific LoRA parameters via distribution matching (forward-KL against context-conditioned references); and (2) a cross-session consolidation stage employing a lightweight recurrent gating mechanism that learns task-specific integration policies for merging successive session memories into a fixed-size state. The key claimed contribution is "lifecycle independence": accumulated memory persists and remains usable when the serving backbone is replaced, requiring only adaptation of the model-specific decoder while preserving the encoder, recurrent gate, and consolidated memory state. Evaluation is conducted across three long-term memory benchmarks (PERMA, PersonaMem-v2, PrefEval) and five diverse backbones, reporting strong empirical results, comprehensive ablations validating both architectural components, and detailed analyses of learned consolidation dynamics revealing domain-structured memory organization.

Critical strengths
  1. This paper demonstrates that RPMem achieves 85.52% on PERMA, outperforming the strongest parametric baseline Metis-9B by 5.32 percentage points and the strongest textual baseline Full Context by 12.98 percentage points. The evaluation spans three distinct benchmark paradigms (event-driven preference tracking in PERMA, long-horizon personalization in PersonaMem-v2, and preference retention over extended intervening turns in PrefEval), showing that the approach generalizes across different memory-demand formulations.
  2. This paper provides insightful analysis of learned dynamics. It shows that RPMem writes more information when a new domain emerges, uses similar write patterns for events in the same domain, and better preserves memories from different domains. After 60 updates, emergence sources retain 6.64× more memory than supplements, suggesting that RPMem learns meaningful memory update strategies.
  3. This paper provides thorough ablation studies isolating design choices.
Critical weaknesses
  1. Although RPMem preserves accumulated memory across backbone replacements, transferring it to a new backbone still requires training a target-specific decoder. When backbone models are updated frequently or the candidate model pool changes rapidly, the cumulative adaptation cost could become substantial. Compared with non-parametric memory systems, it remains unclear whether RPMem's reduction in inference overhead can offset the additional training costs, as the paper provides no experiments quantifying this trade-off under different backbone replacement frequencies.
  2. The motivation for RPMem could focus more on the inference efficiency of cross-session memory access, rather than the limitations of retrieval-based memory alone. Non-parametric systems can also improve evidence coverage through iterative retrieval, although this may require additional tokens and model calls. Comparing RPMem with stronger retrieval baselines under similar task performance would better demonstrate whether parametric memory provides meaningful efficiency advantages.
  3. The paper could be improved in terms of clarity and readability. In Section 4, some key terms and concepts used to describe the experimental design are not sufficiently explained in the main text and require readers to consult the appendix.
  4. For a trained parametric memory, an important question is whether its memory writing and reading capabilities generalize to unseen data distributions. For example, can RPMem trained on PERMA and PrefEval be directly applied to PersonaMem-v2 without further training or adaptation? Evaluating such cross-benchmark transfer would provide stronger evidence of the generality of the learned memory representations.
Questions for authors

Overall, I consider this a borderline paper that could reasonably be either accepted or rejected. I look through parts of the appendix, and I have one main question: Do you think the comparison with Metis is entirely fair in terms of the data used? Why not evaluate Metis and RPMem on the LoCoMo dataset?

9 comments
AuthorsOpenReview ↗

Response to Reviewer WWZQ (1/3)

Thank you for the careful review. We address the concerns about deployment cost, cross-session access, experimental clarity, transfer, and Metis comparison below.

W1: Amortized Cost under Backbone Replacement

Thank you for raising the question of whether the cost of decoder adaptation under frequent backbone replacement could outweigh RPMem's inference savings. We have added an amortized lifecycle-cost analysis covering decoder adaptation, online memory writing, and querying across different user populations and backbone replacement frequencies. In RPMem, a new serving backbone requires adapting its model-specific decoder once, shared by all users; the session encoder, recurrent gate, and accumulated user memories are retained. Existing memories remain available, and online writes and queries require forward computation rather than retraining. The target Qwen3.5-9B decoder takes 30.92 hours on eight GPUs (247.36 GPU-hours) to adapt (original Appendix G.2, Table G.3).

To quantify the trade-off, we compare RPMem with single-pass LightMem, a competitive non-parametric baseline, using measured PrefEval 300-turn query times: 146.24 ms and 326.88 ms per query, respectively. RPMem's measured session-write time is 0.043 s in the 64-session streaming profile. LightMem takes 177.63 s to ingest a 301-session PrefEval history, or 0.590 s/session amortized over that history. We use these measurements to model one year with one write per user per day, assigning each newly supported backbone the measured 247.36 GPU-hour adaptation cost. Online timings are converted to single-GPU-equivalent hours.

Table 1. Break-even memory queries per user per day across backbone replacement frequencies and user populations.

Replacement frequency1K users10K users100K users1M users
Annually10.480.000.000.00
Quarterly51.002.370.000.00
Monthly159.0413.180.000.00
Weekly699.2867.204.000.00

Each threshold is the daily query frequency above which RPMem has lower annual adaptation-plus-online cost. Zero means writing savings alone cover adaptation. Even with weekly backbone replacements, the threshold falls from 67.20 queries/user/day at 10K users to 4.00 at 100K users.

Table 2. One-year cost for 100K users, 10 queries and one write per user per day (GPU-equivalent hours).

Replacement frequencyRPMem totalLightMem totalNet savingReduction (%)
Annually15,510.4439,123.9423,613.5060.36
Quarterly16,252.5239,123.9422,871.4258.46
Monthly18,231.4039,123.9420,892.5453.40
Weekly28,125.8039,123.9410,998.1428.11

Under this workload, RPMem reduces estimated cumulative computation by 53.40% with monthly and 28.11% with weekly replacements, as shared decoder adaptation is amortized across users.

AuthorsOpenReview ↗

Response to Reviewer WWZQ (2/3)

W2: Motivation and Iterative Retrieval

Thank you for highlighting the importance of inference efficiency in cross-session memory access and the need for a stronger iterative-retrieval comparison. To address these points, we clarify the efficiency motivation behind RPMem and extend our evaluation to include a two-pass LightMem baseline. In long-running agents, accumulated information may be needed repeatedly across future queries. RPMem consolidates this information into a fixed-size latent state decoded into LoRA parameters, avoiding repeated retrieval and processing of memory text.

Non-parametric memory preserves inspectable, source-linked records. Iterative retrieval can improve access to this evidence but adds query computation. We therefore evaluated two-pass LightMem alongside the single-pass setting introduced in W1, on the same 540 PrefEval 300-turn questions.

Table 3. Query accuracy and efficiency on PrefEval (300-turn).

MethodAccessAccuracy (%)Latency (ms)LM tokens/query
RPMemParameter-space75.19146.24112.39
LightMemSingle-pass retrieval74.81326.88566.35
LightMemTwo-pass retrieval75.741,518.251,116.85

The second retrieval pass improves LightMem accuracy from 74.81% to 75.74%. RPMem reaches 75.19%, within 0.55 percentage points (pp) of two-pass LightMem, with a 10.38× query-speed advantage and 9.94× lower LM token use. Against single-pass LightMem, RPMem achieves slightly higher accuracy and 55.26% lower query latency.

W3: Clarity of Section 4

Thank you for pointing out that Section 4 relies too heavily on the appendix to explain its key concepts. We have revised this section to define these concepts and clarify the analytical setup directly in the main text. The opening now explains task-conditioned history replay and aggregation at the held-out-user level. Section 4.1 defines domain emergence and supplement, and explains write magnitude, write-pattern similarity, and one-step source survival, including how multiple compiler segments are combined. Section 4.2 distinguishes the survival of a session's contribution in the recurrent state from functional retention, measured by answer accuracy after the target event and later intervening sessions. The appendix retains detailed statistical procedures.

W4: Cross-Benchmark Generalization

Thank you for raising the question of whether RPMem's learned consolidation behavior generalizes to an unseen benchmark without retraining. We address this through a zero-shot transfer experiment from PERMA to PersonaMem-v2. We transferred 70 PERMA-trained consolidation gates (seven variants × ten leave-one-user-out folds) to PersonaMem-v2 using RPMem with Qwen3.5-9B. The compiler, decoder, and gates were frozen. Each gate was tested on all 5,000 PersonaMem-v2 questions, and accuracies were averaged across folds within each PERMA variant and then equally across the seven variants.

Table 4. Cross-benchmark transfer to PersonaMem-v2 (accuracy, %).

MethodConsolidation/training sourceAccuracy
Metis-9BReleased checkpoint29.64
RPMem, latent averagingNone35.25
RPMem, recurrent gatePERMA (zero-shot)38.14
RPMem, recurrent gatePersonaMem-v245.73

Without PersonaMem-v2 gate training, RPMem reaches 38.14%, exceeding latent averaging (35.25%) by 2.89 pp and released Metis-9B by 8.50 pp. Target-task training further improves accuracy to 45.73%.

AuthorsOpenReview ↗

Response to Reviewer WWZQ (3/3)

Q1: Metis Comparison Fairness and LoCoMo

Thank you for raising the question of whether differences in downstream supervision affect the fairness of the Metis comparison, and for suggesting an evaluation on LoCoMo. We address these points through two complementary comparisons—zero-shot transfer and matched target-task adaptation—and an additional LoCoMo (Gold) evaluation.

First, the zero-shot transfer experiment in W4 compares PERMA-trained RPMem with released Metis-9B on PersonaMem-v2, where neither receives target-benchmark adaptation; RPMem leads by 8.50 pp. Second, we adapted Metis with the same downstream task supervision as RPMem on PERMA Noisy SD: ten leave-one-user-out splits, five epochs, answer-label supervision, and 705 evaluation questions. We trained Metis's native memory Writer parameters across eight blocks and rank-5 LoRA on writing-side key/value projections, freezing the backbone and memory Reader. Gradients passed through the Reader to the Writer. This uses 507,920 trainable parameters, versus 524,800 for RPMem's recurrent gate. Both use Qwen3.5-9B-family backbones.

Table 5. Supervised adaptation on PERMA Noisy SD (accuracy, %).

MethodAdapted componentTrainable paramsAccuracy
Metis-9BNone (released)080.47
Metis-9BMemory Writer507,92087.00
RPMemRecurrent gate524,80096.50

Metis improves from 80.47% to 87.00% (+6.53 pp) with target-task adaptation; RPMem reaches 96.50%, retaining a 9.50 pp advantage under aligned downstream training and evaluation.

To evaluate persistent memory for long-running agents, PERMA, PersonaMem-v2, and PrefEval respectively test preference formation/revision, changes in user information, and retention through intervening dialogue; separate cross-backbone experiments assess lifecycle continuity. LoCoMo (Maharana et al., 2024) focuses on factual QA grounded in dialogue evidence, including multi-hop and temporal questions. Metis also reports using entities from LoCoMo as seeds for synthetic training data, while stating that its original QA behavior was not used (Metis, Sec. 6.1); we prioritized benchmarks without this source overlap.

We additionally evaluated RPMem in the LoCoMo (Gold) evidence-session setting used by Metis.

Table 6. LoCoMo (Gold) accuracy (%).

MethodMemory accessAccuracy
RPMem (Qwen3.5-9B)Parametric16.99
Metis-9BParametric16.81
RAG (top-5)Retrieved text21.97
Full ContextGold-session text63.00

RPMem: our new evaluation; other results: Metis, Table 6.

RPMem reaches 16.99%, close to Metis-9B (16.81%), while RAG (top-5) and Full Context achieve 21.97% and 63.00%, respectively. This is consistent with the analysis in Metis (Zhang et al., 2026, Sections 6.2 and 8): direct access to original evidence benefits factual QA, while compressing history into a fixed-size latent state can lead to information loss and confusion between semantically similar facts. These observations highlight the complementary value of explicit textual evidence for factual questions.

We have incorporated the new experiments and cost analysis, emphasized cross-session access efficiency in the introduction, and clarified the Section 4 definitions in the revision.

References

[1] Zhang, Z., et al. (2026). Metis: Memory Foundation Model. arXiv:2607.26760. https://arxiv.org/abs/2607.26760

[2] Maharana, A., et al. (2024). Evaluating Very Long-Term Conversational Memory of LLM Agents. Proceedings of ACL 2024. https://aclanthology.org/2024.acl-long.747/

Reviewer_P9xrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RPMem, a parametric memory approach that decouples persistent memory from the serving language model. Unlike approaches such as Metis, whose memory is tied to a specific backbone, RPMem encodes each session into a shared latent space and uses a learned recurrent gate to consolidate it into a fixed-size memory state. This way, PRMem can easily adapt to new backbone model, providing greater flexibility. Compared with text-based memory, RPMem provides a fixed-size representation that does not occupy the input context, although its portability requires decoder training. Evaluations across three long-term memory benchmarks and five backbones demonstrate competitive performance, cross-backbone memory reuse, and near-constant memory footprint and per-session update cost as the number of sessions grows.

Critical strengths

Strength:

  • Modular design supports backbone replacement. Separating persistent memory from the backbone-specific decoder allows accumulated memory, the session encoder, and the consolidation gate to be retained when changing the serving model.
  • The context-to-adapter distillation objective trains the session resampler and LoRA decoder to reproduce context-conditioned behavior while keeping the backbone frozen. This separates memory compilation from modification of the backbone’s original parameters.
  • Recurrent gate works here too, to support accumulation.
Critical weaknesses

Weakness:

  • it is not a true decoupling? The compilation objective remains tied to a particular backbone’s behavior, and switching backbones requires training a new decoder.
  • RPMem’s consolidation gate receives benchmark-specific training, whereas the released Metis checkpoint is evaluated directly. Together with the different backbones, this makes it difficult to attribute the performance gap specifically to the proposed architecture.

The evaluation shows clear benefits from the approach. My main considerations is simply that the approach is kinda stuck in between of "customized to model" (Metis like approach) vs "completely agnostic" (input-based memory). It still requires model adaptation, yet not taking advantaged of model architecture.

Questions for authors
  1. I'd like a bit clarificaiton on the scope of model agnostic aspect.
  2. The comparison with Metis, in terms of the second stage "adaptation" training.
5 comments
AuthorsOpenReview ↗

Response to Reviewer P9xr (1/2)

We thank the reviewer for the careful evaluation and constructive feedback. We address the scope of backbone independence and the fairness of downstream adaptation below.

W1 & Q1: Scope and Significance of Backbone Independence

Thank you for highlighting the need to clarify the scope of RPMem's backbone independence, given that memory compilation is conditioned on the serving model and backbone replacement requires a new decoder. We clarify that backbone independence concerns the lifecycle of persistent memory: accumulated memory can be preserved and continuously updated across backbone replacements.

This addresses a practical requirement of long-running LLM agents. Backbone-integrated parametric memory, such as Metis, ties its memory state to the model that maintains it. Text-based memory can be reused across models, but accessing that memory requires retrieving and processing stored text at query time. RPMem combines persistent memory portability with parameter-space memory access. It consolidates successive sessions into a fixed-size latent state, which is decoded into LoRA parameters that inform the backbone's computation without historical-text retrieval or context injection.

The model-specific decoder serves as the interface between the persistent memory state and each serving backbone. A decoder is adapted once for each newly supported backbone and shared across users. The accumulated memories, session encoder, and recurrent consolidation gate remain unchanged, allowing memory to persist and evolve across backbone upgrades without replaying historical sessions.

AuthorsOpenReview ↗

Response to Reviewer P9xr (2/2)

W2 & Q2: Downstream Adaptation and Comparison with Metis

Thank you for raising the question of whether Stage II's downstream supervision affects the fairness of the comparison with the released Metis checkpoint. RPMem separates Single-Session Memory Compilation, which learns session representations through context-conditioned distribution matching, from Cross-Session Memory Consolidation, which trains a recurrent gate to retain, revise, and integrate information under downstream supervision. We examine the role of this supervision through two complementary experiments: cross-benchmark zero-shot transfer and target-task adaptation of Metis.

Experiment 1: Cross-benchmark transfer.

Using RPMem with Qwen3.5-9B, we transferred 70 PERMA-trained gates (seven variants, ten leave-one-user-out folds each) to PersonaMem-v2. The memory compiler, decoder, and gates were frozen. Each gate was evaluated on all 5,000 questions, with accuracies averaged across folds within each source variant and then equally across the seven variants.

Table 1. Cross-benchmark consolidation on PersonaMem-v2 (accuracy, %).

MethodBackboneConsolidationTraining sourceAccuracy
RPMemQwen3.5-9BLatent averagingNone35.25
Metis-9BQwen3.5-9BNative recurrent memoryReleased checkpoint29.64
RPMemQwen3.5-9BRecurrent gatePERMA38.14
RPMemQwen3.5-9BRecurrent gatePersonaMem-v245.73

The PERMA-trained gate achieves 38.14%, exceeding latent averaging (35.25%) by 2.89 percentage points (pp) and Metis-9B (29.64%) by 8.50 pp, without PersonaMem-v2 supervision. Training the gate on PersonaMem-v2 further raises accuracy to 45.73% (+7.59 pp).

Stage II learns transferable consolidation behavior, while target-task supervision further adjusts what information is retained, revised, and integrated according to the application's prediction requirements.

Experiment 2: Target-task adaptation of Metis.

We next examine the comparison when Metis also receives target-task supervision. We adapted the released Metis-9B checkpoint on PERMA Noisy SD, using the same ten leave-one-user-out splits, five training epochs, answer-label supervision, and 705 evaluation questions as RPMem.

We trained Metis's memory Writer across all eight memory blocks, updating its native writing parameters and rank-5 LoRA on writing-side key/value projections. The backbone and memory Reader were frozen; answer-label gradients propagated through the Reader to the Writer. Rank 5 gives 507,920 trainable parameters, close to the 524,800 parameters of RPMem's recurrent consolidation gate. Both methods use Qwen3.5-9B-family backbones.

Table 2. Target-task adaptation on PERMA Noisy SD (accuracy, %).

MethodBackboneAdapted componentTrainable parametersAccuracy
Metis-9BQwen3.5-9BNone (released)080.47
Metis-9BQwen3.5-9BMemory Writer507,92087.00
RPMemQwen3.5-9BRecurrent consolidation gate524,80096.50

Target-task adaptation improves Metis from 80.47% to 87.00% (+6.53 pp). RPMem achieves 96.50%, maintaining a 9.50 pp advantage under the same downstream data, evaluation protocol, and training epochs, with trainable parameter counts differing by 3.2%.

These results support RPMem's advantage under both zero-shot transfer and matched downstream supervision. We have incorporated both experiments and their protocols into the revised appendix.

Dear Reviewer P9xr,

As the discussion period is nearing its end, we would like to kindly follow up and ask whether you have had a chance to review our responses and additional experiments. We would be grateful if you could let us know if they have addressed your concerns or if any further questions remain. We would be happy to provide any additional clarification before the discussion window closes.

Thank you again for your time and constructive feedback.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Summary

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have provided detailed point-by-point responses and revised the manuscript accordingly. All changes are highlighted in red in the revised PDF.

The main revisions are summarized below:

  1. Improved motivation and clarity
    • In the Introduction, we have emphasized efficient cross-session memory access as a key motivation for RPMem.
    • In Section 2.2, we have clarified the roles of the frozen context encoder and trainable resampler.
    • In Section 4, we have refined the definitions and experimental descriptions of memory dynamics to improve readability.
  2. Additional comparisons with related methods
    • In Appendix D.5, we have added comparisons with SHINE, both with and without recurrent consolidation.
    • In Appendix H.3, we have included a target-task adaptation experiment for Metis-9B under matched downstream supervision and comparable trainable parameter budgets.
    • In Appendix H.4, we have added a LoCoMo (Gold) evaluation to complement the main benchmarks.
  3. Extended generalization and lifecycle evaluation
    • In Section 3.3 and Appendix D.6, we have added zero-shot transfer experiments evaluating PERMA-trained consolidation gates on PersonaMem-v2, including a backbone-matched comparison with textual baselines.
    • In Section 3.4 and Appendix G.3, we have evaluated continued memory updates after backbone replacement, retaining the accumulated memory and original consolidation gate.
  4. Expanded efficiency analysis and reproducibility
    • In Appendix E.2, we have clarified baseline implementation details and documented the specific versions of Mem0 and LightMem.
    • In Appendix E.8, we have compared RPMem with single-pass and two-pass LightMem in terms of accuracy, query latency, and token consumption, and detailed the two-pass retrieval procedure.
    • In Appendix E.9, we have added an amortized deployment-cost analysis across different user populations and backbone replacement frequencies.
  5. Additional diagnostic experiments
    • In Appendix I.1, we have included a Relevant Context evaluation to investigate Full Context performance degradation.
    • In Appendix J.1, we have added a controlled preference-intervention case study to examine the functional retention and retrieval of specific information.

We thank the reviewers and Area Chair again for their time and consideration, and welcome any further feedback.

AuthorsOpenReview ↗

Discussion Summary

Dear Area Chair and Reviewers,

Thank you again for the careful reviews and constructive discussion throughout the review period. We appreciate the reviewers' recognition of several key strengths of RPMem: its modular two-stage design that decouples persistent parametric memory from backbone-specific LoRA decoding to support memory reuse across model replacements (P9xr, oQ9e), strong empirical performance across three distinct long-term memory benchmarks (WWZQ, oQ9e), thorough ablations and practical efficiency with near-constant memory footprint and update cost (WWZQ, oQ9e), and insightful analyses of learned consolidation dynamics (WWZQ).

Following the discussion, all three reviewers recommend acceptance: Reviewer oQ9e noted that the additional experiments substantially strengthen the paper and maintained Clear Acceptance, Reviewer WWZQ raised their rating to Weak Acceptance with no further questions, and Reviewer P9xr highlighted the matched Metis adaptation evaluation and reaffirmed their vote for acceptance. To address every question raised during the discussion, we added controlled comparisons with SHINE (Appendix D.5), zero-shot cross-benchmark transfer to PersonaMem-v2 including backbone-matched textual baselines (Section 3.3, Appendix D.6), continued memory updates after mid-history backbone replacement (Section 3.4, Appendix G.3), target-task adaptation of Metis-9B and LoCoMo (Gold) evaluation (Appendices H.3–H.4), two-pass LightMem efficiency and amortized lifecycle deployment-cost analyses (Appendices E.8–E.9), diagnostic studies (Appendices I.1, J.1), and explicit baseline implementation and version details (Appendix E.2). All revisions have been incorporated into the updated PDF (highlighted in red) and indexed in our Revision Summary.

We thank the Area Chair for considering these revisions and the reviewers' updated assessments in reaching a final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.