acceptodds
Under review as a conference paper at ICLR 2027

Some Memories Must Fade: Persistence and Decay of User Memories in Model Parameters

Abstract

Parametric memory allows Large Language Models (LLMs) to internalize knowledge directly into their weights, offering a potentially powerful alternative to retrieval-based memory (RAG). However, existing approaches to parametric memory are difficult to deploy as they largely mirror the Continual Learning (CL) objective of retaining information indefinitely. This is neither sustainable nor sufficient for a memory system, which must govern knowledge across its lifecycle by retaining stable facts, reinforcing recurring observations, updating changed beliefs, and allowing transient information to fade. We propose a system for parametric memory management consisting of a memory substrate (handling information storage and forgetting) and a learned policy (what to remember, how deeply, and for how long). Our substrate is a fixed-capacity LoRA adapter that stores memories in sparse k-hot combinations of adapter ranks to limit interference. This allows each rank and its stored information to maintain an independently controlled temporal decay, enabling outdated information to fade while long-term facts persist. On this substrate, we train a lightweight 2M-parameter controller that governs each incoming user event to ensure the memory system reflects the user's true, updated belief state. To evaluate how memory systems manage user memory, we introduce a controlled dataset of long-horizon user trajectories with varying persistence of facts based on changing belief states and user-behavioral shifts, with probe queries that test recall while information is valid and abstention after it expires. Our policy learns to effectively manage user memory over its entire lifecycle, balancing recall and abstention to achieve 86% of oracle performance, unlike CL and RAG baselines that primarily default to one of the two behaviors. By providing the first system to manage memory directly in model weights, this work establishes a foundation for long-horizon parametric memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parametric memory, Memory management, Selective memorization, Controlled forgetting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CSKZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a parametric memory management system, which aims to handle different requirements of retaining information in memory. For instance, some information is not valid after a short period of time and thus should be forgotten (e.g., “I had a bagel for breakfast should not remain valid even the next day”), but some should be retained much longer (e.g., “I work as a teacher should likely be relevant information for years”). The proposed system consists of two conceptual components, a controller to learn and enforce the memory retention policy and the LoRA-based memory substrate. Since real-user trajectories are not easily available, the authors essentially designed a simulator to create different types of memory tasks, as well as the dataset for evaluation. Experiments show that the proposed method balances the accuracy of the retrieved memory value (VAL) and when the system should abstain (ABS), compared with RAG and CL (continual learning) methods.

Critical strengths
  • The motivation of this work is convincing -- addressing the needs of having different information retention policy with parametric memory
  • The system design is novel -- while it borrows some existing components (e.g., FlyLoRA and LoRA-FA), they are somewhat repurposed and combined in a new way. Learning the policy for parametric memory also seems new.
Critical weaknesses
  • The evaluation is done using only simulated scenarios and data. How effective the proposed design is to real-world problems still remains unknown. (The authors do state that the dataset is “a controlled stress test” and acknowledge the coverage is limited. In addition, Appendix D studies the effects of using natural language queries instead of templated ones, which shows a large performance drop.)
Questions for authors
  • Should long-lived (STABLE) facts have expiration dates? Looking at Table 10, the “unmanaged d” baselines achieved much higher scores on VAL but low ABS scores (as they don’t easily forget).
  • What are the results of trivial baselines (e.g., always abstain and always answer)?
  • The ground-truth validity is generated by a fixed decay-horizon rule (App. C, Fig. 4). How would results change if validity were determined by a different process (e.g., invalidated by a different event)?
Reviewer_Tk1MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how facts stored in an assistant’s parameters should persist, be reinforced, change, or expire. It proposes a rank-32 LoRA memory with sparse two-rank routing and per-rank decay, a learned controller that decides whether and how strongly to write each event, and a separate retrieval adapter for answering or abstaining. On synthetic user trajectories, the system with learned routing achieves 0.56 class-balanced accuracy, compared with 0.50 for the reported retrieval-based memory manager and 0.42 for the strongest continual-learning baseline. Perfect routing raises the score to 0.60, while a controller with privileged lifecycle metadata reaches 0.65.

  • The lifecycle formulation and diagnostics are valuable, but the unclear implementation, weak retention of stable facts, and limited conversational evidence prevent a weak accept. I would be happy to reconsider my score if the authors address these concerns in their response.
Critical strengths
  • The paper poses a useful memory-management problem. Its evaluation considers valid recall, expiration, reinforcement, and revisions rather than retention alone.

  • Sparse writes and per-rank decay provide concrete controls over interference and forgetting. Removing per-rank lifespan control lowers the perfect-routing system’s accuracy from 0.60 to 0.48.

  • The comparisons among learned routing, perfect routing, and the metadata-informed controller help identify different sources of error. The per-class results also expose important trade-offs.

Critical weaknesses

The network architecture and retrieval path are not specified clearly enough to reproduce. The paper does not identify the Retrieval LoRA’s target layers and projections or explain precisely how it combines with the routed Memory LoRA. It is also unclear whether rank masks are shared across layers. The reported retrieval-training setting is inconsistent: p_unk(20) is 0.9 in Section 3.3 but 0.38 in Appendix A.3.

Long-term stable facts remain a major failure case. With learned routing, valid-query accuracy for stable facts is only about 0.34. Later short-lived writes can overwrite the decay rate of a shared rank and cause an older fact to fade prematurely.

The main evaluation relies on generated trajectories, masked values, and templated queries. In the appendix’s natural-language query experiment, class-balanced accuracy is 0.444 even with the metadata-informed controller. The results do not yet establish effectiveness in ordinary conversations.

Questions for authors
  • Could the authors provide a clear, end-to-end description of the model architecture and execution flow for both writing a memory and answering a query? Please specify which parameters and adapters are loaded during each process, which are active, frozen, or updated, and how the backbone, router, controller, Memory LoRA, and Retrieval LoRA interact throughout the forward and update steps. The current description does not make the actual computation clear enough to evaluate or reproduce the method.

  • What are the measured end-to-end compute cost and latency of processing a new memory, including the 20–150 optimization steps, compared with updating the retrieval-based memory system?

  • How should the system answer a time-qualified question about an expired but historically true fact? The current benchmark rewards abstention after expiry, whereas that question requires the earlier value.

3 comments

We wanted to follow up on our rebuttal to check if you have any remaining concerns. We would be happy to provide any further clarifications/details required. We would also appreciate if you could consider updating your recommendation if your concerns are resolved. Thank you for taking the time to review our work.

Thank you very much for the detailed response and the substantial work put into the revision.

I find the research question valuable and the overall approach promising. The response has also clarified many of my earlier questions.

My main remaining concern is the presentation of the manuscript. For example, Figures 2 and 3 would benefit from a more considered visual design, and the architectural details, execution flow, and empirical trade-offs could be communicated more coherently in the paper itself. I encourage the authors to give the manuscript a thorough editorial and visual revision.

Thank you for your positive assessment of the research question and the proposed approach.

We take your concern regarding the presentation seriously and thank you for your feedback. In addition to the changes made to the manuscript after the initial review, which included a dedicated description and a figure explaining the execution flow, we have now added a high-level description of the complete architecture at the start of Section 3 (lines 195–201). We are also looking at revising the figures to communicate the contributions more clearly.

The proposed architecture consists of several interdependent components. Our current structuring aims to describe each component and the reasoning behind its design before moving on to the next component, bringing them together in Section 4.3. We recognize that an earlier overview would help readers better understand how these components interact and have attempted to address this through the summary at the beginning of Section 3.

In the current version, we explicitly discuss several empirical trade-offs, including memorization versus abstention (lines 411-415), varying retention requirements across memory classes (lines 424-430), lifespan versus compression (lines 431-448, Table 4), and read latency versus write cost (§4.5, Table 10).

We also welcome any further specific feedback on aspects of the presentation that we can improve. We hope that the discussion and subsequent revisions have addressed your concerns and would appreciate your reconsideration of the overall recommendation. Your insights have been valuable, and we look forward to incorporating your remaining feedback.

Reviewer_9YBrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies user memories that should persist, expire, accumulate reinforcement or be superseded. A sparse LoRA substrate combines write depth and per-rank decay, while a learned controller chooses memory-management actions. A synthetic trajectory benchmark supplies lifecycle supervision. The problem formulation is compelling, and the proposed mechanism follows it consistently. The main uncertainty concerns how much the experiments establish about parametric storage specifically.

Critical strengths
  1. The paper asks a useful question that conventional retention-focused evaluation can miss.

  2. Separating valid retrieval from appropriate abstention makes the lifecycle tradeoff visible.

  3. Substrate measurements, controller ablations and analysis of stable-fact failures provide meaningful insight.

  4. The inclusion of a write-time managed RAG baseline is welcome.

  5. Appendix D tests natural-language retrieval while acknowledging that the strongest lifecycle results use a templated interface.

Critical weaknesses

The comparisons do not yet isolate the advantage of storing memories in parameters. The proposed controller receives lifecycle supervision through imitation learning, whereas the managed RAG baseline uses a prompted manager. A similarly trained external-memory controller with explicit expiry would provide a stronger test.

Evaluation uses only 15 held-out users. Query-level Wilson intervals do not account for correlation within a user’s trajectory, which matters when interpreting the 0.56 versus 0.50 class-balanced scores.

Decay also acts on shared rank directions: combinatorial addresses are not independent storage slots, and later writes can alter other memories’ persistence. The substantial cost to stable-fact retrieval deserves emphasis.

Natural-language transfer remains limited. Appendix D reports 0.444 class-balanced accuracy using the metadata-oracle controller, rather than the complete learned pipeline. Matching generator size alone also does not establish compute parity when writes require repeated gradient steps.

Questions for authors
  1. What happens when an external-memory baseline receives the same lifecycle supervision and controller-training budget?

  2. Do the gains survive uncertainty estimates clustered by user and repeated controller-training seeds?

  3. How does the learned controller perform end to end with natural-language writes and queries, including stable-fact retention and measured write/query cost?

Comments on the paper

2 comments

Dear reviewers Thank you for your thoughtful review. The authors have submitted their rebuttal, and the discussion period is now open. Please take a moment to read the authors' response and engage with them and your fellow reviewers to address any remaining questions or clarify key points. Your active participation during this phase is vital to reaching a fair and well-informed final recommendation.

Best regards, area chair

AuthorsOpenReview ↗

Summary of the Revisions

We briefly summarize the main concerns addressed in our revision:

Our primary contribution is an architecture enabling parametric memory management (the substrate), alongside a learned policy that accounts for different requirements (selective storage, retention, updating, and forgetting) of memory systems.

Natural-language transfer (all reviewers)

While our initial evaluation used templated queries to isolate memorization from management, our rebuttal includes end-to-end trained natural language experiments achieving comparable performance. Reviewer 9YBr acknowledged their concern regarding this had been addressed.

Stable retention (all)

Reviewers had raised concerns about the lower performance of facts which should be preserved indefinitely/for long horizons. We implemented the mitigation strategy identified in our initial submission by allocating dedicated storage ranks for long-term facts, more than doubling the retention of long-term facts. We also discuss the corresponding retention-abstention trade-off in the revision, as suggested by Reviewer 9YBr.

RAG comparison and efficiency (9YBr, Tk1M)

We added trained RAG memory manager baselines that match the performance of our parametric substrate, demonstrating its competitiveness in performance, as well as a latency analysis which aligns with the expected faster reads but slower writes for parametric memory.

Presentation (Tk1M)

We have included a dedicated description of the end-to-end execution flow (Section 4.3, Figure 3) and a high-level overview at the beginning of Section 3 to allow for easier understanding of the contribution. As per the reviewer's suggestion, we are further refining the figures and identifying areas we can present more clearly.

Trivial baselines and synthetic data (CSKZ)

We added always-abstain and always-latest-value baselines, and identified the limitations of using only aggregate accuracy to interpret results. We additionally clarified why controlled synthetic data with ground truth is required for our setting, with similar user data collection restrictions as previous works on LLM memory.

Additional details

We also added user-clustered CIs and multi-seed results, along with clarifications and more detailed write-ups for the proposed architecture.

With all additions incorporated into the revised manuscript, we believe we have addressed the major concerns raised by reviewers. Reviewer 9YBr acknowledged that their main concerns were largely resolved, while Reviewer Tk1M acknowledged that many of their earlier questions were clarified, identifying presentation as the main remaining concern. We have since further revised the explanation and will continue refining the presentation.

Thank you for your time and consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.