- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes RPMem, a parametric memory framework for LLM agents. It first trains a session encoder (consisting of a frozen context encoder and a trainable resampler) and a decoder to compress individual interaction sessions into latent representations that are mapped to LoRA weights. A second training stage learns a recurrent aggregation mechanism that combines memories across sessions into a fixed-size representation. The accumulated memory can also be reused across different LLM backbones by adapting only the decoder. Experiments on three long-term memory benchmarks show improvements over existing text-based and parametric memory methods, with approximately constant memory size and update cost.
Critical strengths
-
Interesting combination of existing techniques: The paper combines context-to-LoRA compilation, previously explored by Doc-to-LoRA [1] and SHINE [2], with incremental latent memory updates, previously explored in approaches such as MemoryLLM [3] and M+ [4]. RPMem introduces a learned recurrent consolidation mechanism and separates accumulated memory from backbone-specific LoRA decoding, enabling memory reuse across model replacements.
-
Strong empirical results and efficiency: RPMem achieves competitive or superior performance against existing text-based and parametric memory methods across three benchmarks. The ablations show substantial benefits from learned consolidation, while the approximately constant memory footprint and update cost suggest practical advantages for long-running agents.
[1] Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, and Robert Tjarko Lange. Doc-to-LoRA: Learning to instantly internalize contexts. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=iW1oBBO72S.
[2] Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelberg, Haggai Maron, and Muhan Zhang. SHINE: A scalable in-context hypernetwork for mapping context to LoRA in a single pass. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026c. URL https://openreview.net/ forum?id=ZMexYcAibv.
[3] Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, and Julian McAuley. MEMORYLLM: Towards selfupdatable large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 50453–50466. PMLR, 2024c. URL https://proceedings.mlr.press/v235/wang24s.html.
[4] Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, and Zexue He. M+: Extending MemoryLLM with scalable long-term ´ memory. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 63308–63323. PMLR, 2025a. URL https://proceedings.mlr.press/v267/wang25au.html.
Critical weaknesses
I find the paper technically strong and the results promising. While the following limitations do not change my overall positive assessment, addressing them would help strengthen the novelty claims and practical motivation.
-
Incomplete comparisons with closely related methods. The paper does not experimentally compare against Doc-to-LoRA [1], SHINE [2], or several existing latent-memory approaches (e.g., M+ [4]). This is particularly concerning because context-to-LoRA compilation (e.g., Doc-to-LoRA [1] does that) and recurrent latent memory updates (e.g., M+ [4] maintains a fixed-size pool of latent memory tokens within the transformer) have already been explored independently in other works. A comparison with Doc-to-LoRA [1] or SHINE [2] extended with recurrent aggregation would help establish whether RPMem's architecture provides meaningful improvements beyond combining existing techniques. Ideally, also compare recurrent consolidation with M+ [4], but I understand that it might be infeasible during the rebuttal as it would require significant architectural changes. These comparisons would help better isolate the contribution of RPMem's particular architecture beyond combining existing techniques.
-
Limited practical benefits of lifecycle independence. Although RPMem preserves the accumulated latent memory across backbone replacements, adapting to each new backbone still requires training a target-specific decoder on 664K compilation sessions (10,377 optimizer updates). The paper demonstrates that reusing the learned encoder outperforms training a new compiler from scratch under the same target-side budget, but does not establish the total computational advantage of this approach relative to lightweight text-based memory, which requires no backbone-specific memory retraining. This leaves the practical significance of the claimed lifecycle independence only partially established.
Questions for authors
-
Missing closely related baselines: Why are Doc-to-LoRA [1], SHINE [2] not included in the experimental comparison? In particular, could the authors evaluate context-to-LoRA methods extended with recurrent aggregation to better isolate the novelty of RPMem?
-
Unexpected Full Context degradation: Why does Qwen3.5-35B-A3B perform substantially worse with Full Context than with No Context in Figure 2? Could the authors verify these results and explain the degradation?
-
Functional validation of latent memory: The memory-dynamics analysis shows correlations between gate coefficients and semantic event types, but does not directly establish which facts are preserved in the latent state. Could the authors conduct controlled experiments with synthetic facts and targeted memory interventions to demonstrate causal retention and retrieval?
-
Generalization of recurrent consolidation: The consolidation gate is trained using task-specific supervision. How well does a gate trained on one benchmark or application generalize to another? Could the authors evaluate cross-task transfer without retraining the gate? This would help establish whether the learned memory-update mechanism is general-purpose or strongly dependent on the downstream task.
-
The central lifecycle claim also involves continuing to update an existing memory after changing backbones. Do the authors explicitly evaluate the sequence: sessions on LLM A -> switch to B -> new sessions on LLM B -> check recall from memory? If your experiments only evaluate decoding an existing memory on a different backbone, that tests portability but not necessarily the full lifecycle scenario.
-
Frozen encoder: The context encoder appears to be frozen while the resampler is trained. Could the authors clarify this distinction earlier in the paper and explain the motivation for freezing the encoder?
7 comments
Response to Reviewer oQ9e (2/3)
Q2: Full Context Degradation on Qwen3.5-35B-A3B
Thank you for highlighting the unexpected performance degradation of Full Context relative to No Context on Qwen3.5-35B-A3B. We have checked the reported results and added a Relevant Context evaluation to examine the effect of unrelated historical information.
Table 3. Qwen3.5-35B-A3B on PERMA (accuracy, %).
| Context condition | Accuracy |
|---|---|
| No Context | 79.48 |
| Full Context | 64.86 |
| Relevant Context | 83.78 |
| RPMem | 94.21 |
No Context, Full Context, and RPMem are the original results (Table E.11); Relevant Context is newly evaluated.
Full Context trails No Context by 14.62 percentage points (pp). Providing relevant history instead yields 83.78%, 4.30 pp above No Context. This pattern is consistent with interference from irrelevant history making it harder to use relevant information in the full context.
Q3: Functional Validation of Latent Memory
Thank you for suggesting a functional evaluation of whether the gate dynamics in Section 4 correspond to the retention and retrieval of specific information. We have added a controlled preference-intervention case study to examine how changing one memory input affects downstream predictions after subsequent memory updates.
Using Qwen3-8B with a trained PrefEval compiler and recurrent gate, we kept all parameters frozen and constructed two histories differing only in one target preference:
- History A: Python learning resources must be accessible to visually impaired users; visually dependent materials should be avoided.
- History B: The user dislikes audio/video learning resources and prefers written explanations and textbooks.
Both histories also contain the same independent preference for manual-transmission rental cars and the same 10 subsequent dialogue turns (20 messages). Each history is compiled and consolidated independently from the same initial state. We then ask two separate multiple-choice questions: “What are the best resources to learn Python programming?” (target) and “Can you recommend a reliable car rental model in Miami?” (control). The reader receives the questions and options, but no history text.
Table 4. Answers after 10 intervening dialogue turns.
| History | Python resources (target question) | Car rental (control question) |
|---|---|---|
| A | Audiobooks for visually impaired learners | Manual-transmission Volkswagen Golf |
| B | Printed Python books | Manual-transmission Volkswagen Golf |
Replacing the learning preference changes the target answer from audiobooks to printed books, while the independent car preference yields the same answer. The pattern holds for both the original and reversed option order. This controlled input intervention connects specific retained information to downstream predictions, complementing the gate-coefficient analysis in Section 4.
Q4: Cross-Task Transfer of the Consolidation Gate
Thank you for raising the question of whether a consolidation gate trained with downstream supervision can generalize to an unseen benchmark without target-task retraining. We examine this through zero-shot transfer of PERMA-trained consolidation gates to PersonaMem-v2. We transferred 70 PERMA-trained gates (seven variants × ten leave-one-user-out folds) to PersonaMem-v2 using RPMem with Qwen3.5-9B. The compiler, decoder, and gates were frozen; each gate was evaluated on all 5,000 target questions. We averaged accuracy across source folds and then equally across the seven source variants.
Table 5. Consolidation transfer to PersonaMem-v2 (accuracy, %).
| Consolidation | Training source | Accuracy |
|---|---|---|
| Latent averaging | None | 35.25 |
| Recurrent gate | PERMA | 38.14 |
| Recurrent gate | PersonaMem-v2 | 45.73 |
Without PersonaMem-v2 gate training, the transferred gate reaches 38.14%, exceeding fixed latent averaging (35.25%) by 2.89 pp. Target-task training further improves accuracy to 45.73% (+7.59 pp), showing the additional benefit of adapting consolidation to the application's memory requirements.
Response to Reviewer oQ9e (3/3)
Q5: Continued Memory Updates after Backbone Replacement
Thank you for highlighting the distinction between preserving an existing memory state and continuing to update that memory after backbone replacement. We have extended the evaluation to test continued memory writing and querying while retaining the accumulated memory and source-trained consolidation gate.
We evaluated this sequence on PERMA Noisy SD, using ten leave-one-user-out folds and 695 questions with multi-session histories. We tested both Qwen3-4B → Qwen3-8B and Qwen3-8B → Qwen3-4B. For each history, we processed the first half of the sessions using backbone A and its source-trained gate (G_A). At the switch, we preserved the accumulated memory state, session encoder, and gate (G_A), replacing only the serving backbone and its adapted LoRA decoder. We then processed the remaining sessions and evaluated the final memory with backbone B.
We compare the retained source gate with a reference that uses the target backbone's trained gate (G_B) throughout the history.
Table 6. Continued memory updates across backbone replacement on PERMA Noisy SD (accuracy, %).
| Backbone transition | Retained source gate (G_A) | Target-trained gate (G_B) |
|---|---|---|
| Qwen3-4B → Qwen3-8B | 85.37 | 91.37 |
| Qwen3-8B → Qwen3-4B | 76.02 | 86.00 |
With the original gate retained, RPMem achieves 85.37% after switching from 4B to 8B and 76.02% in the reverse direction, without retraining the gate or replaying previous sessions. The target-trained gates achieve 91.37% and 86.00%, respectively, showing the additional benefit of target-specific consolidation training.
Thus, adapting only the model-specific decoder supports continued memory updates and querying while preserving accumulated memory and the source gate, consistent with lifecycle independence.
Q6: Frozen Context Encoder and Trainable Resampler
Thank you for pointing out the need to distinguish the training roles of the frozen context encoder and trainable resampler, and to explain the motivation for this design. We have clarified these distinctions in Section 2.2. The session encoder comprises a pretrained, frozen context encoder that extracts token-level features and a trainable Perceiver-based resampler that compresses those features into a fixed-shape latent representation. In Single-Session Memory Compilation, the resampler and model-specific LoRA decoder are trained jointly through context-conditioned distribution matching, while the context encoder and answering backbone are frozen. Cross-Session Memory Consolidation then trains only the recurrent gate, with the compiler frozen.
Freezing the context encoder maintains a stable feature space as serving backbones change. The resampler learns a shared representation for session memories, while each backbone-specific decoder maps that representation to its own LoRA parameters. This allows the same encoder and accumulated memory to be retained when a new backbone is introduced.
We have incorporated the additional experiments and analyses into the revision and clarified the motivation for efficient cross-session memory access and the encoder–resampler training roles in the main text.
Response to Reviewer oQ9e (1/3)
We thank the reviewer for the thoughtful evaluation and constructive suggestions. We address the related-method comparisons and practical value of lifecycle independence first, followed by the questions about evaluation and memory behavior.
W1 & Q1: Related Methods and Recurrent Aggregation
Thank you for suggesting additional comparisons with related methods to clarify RPMem's contribution beyond combining context-to-LoRA compilation with recurrent memory updates. We address this through the existing consolidation ablations and a controlled comparison with SHINE. Our consolidation ablations (Section 3.3) partly isolate this question: Latest Session compiles only the most recent session into LoRA parameters and achieves 53.56% on the four core PERMA settings, versus 85.52% for RPMem. Latent averaging and LoRA factor averaging test fixed cross-session aggregation rules using the same compiler.
We further evaluated the released SHINE-ift_mqa checkpoint on Qwen3-8B across all seven PERMA variants. SHINE w/o Gate uses fixed averaging of compiled session representations. SHINE w/ Gate freezes SHINE's compiler and decoder and adds the same recurrent consolidation gate architecture as RPMem. RPMem w/ Gate uses our fixed-reference forward-KL compiler. Both gates use the same ten-fold leave-one-user-out splits, answer supervision, and five training epochs.
Table 1. Comparison with SHINE on PERMA (accuracy, %).
Adding learned consolidation raises SHINE's seven-variant average from 29.26% to 85.01% (+55.75 percentage points, pp). Under matched gate architecture and downstream training, RPMem improves further by 1.52 pp on the seven-variant average and 1.27 pp on the core average.
Doc-to-LoRA and SHINE learn forward context-to-adapter mappings; M+ maintains updatable latent memory within the serving backbone. RPMem instead preserves a fixed-size recurrent state independently of the backbone and decodes it into model-specific LoRA parameters, allowing accumulated memory to continue across backbone replacements.
W2: Practical Value of Lifecycle Independence
Thank you for highlighting the need to quantify the practical computational cost of lifecycle independence, including decoder adaptation relative to text-based memory. We have added an amortized deployment-cost analysis covering decoder adaptation, online memory writing, and querying across different backbone replacement frequencies.
A new Qwen3.5-9B decoder costs 30.92 hours on eight GPUs (247.36 GPU-hours) to adapt, using 664K compilation sessions and 10,377 optimizer updates. It is trained once per target backbone and shared across users; existing user memory states, the session encoder, and the recurrent gate are retained. We compare with single-pass LightMem on PrefEval 300-turn, using measured query latencies of 146.24 ms (RPMem) and 326.88 ms (LightMem). RPMem takes 0.043 s per session write in its streaming profile; LightMem takes 177.63 s to ingest a 301-session history, averaging 0.590 s/session over the ingestion process.
For a one-year scenario with 100K users, 10 queries and one session write per user per day, we count adaptation for every new backbone and convert online timings to single-GPU-equivalent hours.
Table 2. Annual adaptation, writing, and querying cost (GPU-equivalent hours).
Even after including decoder adaptation, RPMem reduces the estimated annual computation by 53.40% with monthly and 28.11% with weekly backbone replacements.