acceptodds
Under review as a conference paper at ICLR 2027

Learning While Forgetting: Sparse Adapter Consolidation for Memory-Bounded Continual Learning

Abstract

Continual learning under a fixed retained-state budget eventually forces a choice about what to forget. Retaining one adaptation per task grows storage with the stream, while consolidating adaptations into shared memory makes older information progressively harder to recover. We introduce Palimpsest, a write–recover–gate learner that treats consolidation and forgetting as coupled problems. Each task's low-rank adapter is sparsely represented and superposed, with decay, into a shared fixed-size history store. Joint sparse recovery reconstructs candidate historical adapters. A support-recovery analysis motivates an age-dependent reliability horizon, while a calibrated load correction turns this into a deployment forecast that determines when to serve a recovered adapter and when to fall back to the frozen backbone. Reusing the ImageNet-R calibration without refitting predicts the recovery horizon of a running class-incremental learner with . Gate interventions show that fallback recovers 3.1–14.1 percentage points of mean old-task accuracy after the predicted horizon under forced forgetting, while also exposing conservative cases in which useful adaptation is discarded. Under retained-state caps of 5M parameters on ImageNet-R and 15M on OmniBench-1K, Palimpsest leads both reported accuracy metrics at the longest evaluated horizons; at 200 tasks on OmniBench-1K, it exceeds the strongest capped baseline by 7.18 points in average accuracy and 6.90 points in final accuracy. The advantage emerges as memory becomes limiting, supporting sparse consolidation with explicit, forecast-guided forgetting as a principled approach to memory-bounded continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class-incremental learning, memory-bounded learning, parameter-efficient fine-tuning, low-rank adaptation, compressed sensing, sparse recovery, predictable forgetting, pre-trained models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vWf3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies continual learning in a fixed memory constraint setting, focusing on how to retain useful knowledge as new tasks are learned and memory capacity becomes exhausted. The authors propose Palimsest, which compresses task-specific LoRA adapters into a shared, fixed-size memory and approximately reconstructs them when needed. The main idea is to make forgetting predictable by estimating how far back historical adaptations can be reliably recovered, and using that estimate to decide when to fall back to the frozen backbone model. Experiments on image classification benchmarks demonstrate improved accuracy over competing approaches under the evaluated memory constraints, alongside evidence supporting the proposed recovery-horizon predictions.

Critical strengths
  • Interesting and well-motivated problem: The paper addresses continual learning under fixed memory constraints, focusing on what happens when memory becomes saturated. I like the perspective that, rather than attempting to eliminate forgetting, the key idea is that the system should know approximately when it can no longer reliably recover an older adaptation, making forgetting predictable and manageable.

  • Integrating recovery and forecasting: Their combination of sparse coding, shared memory, and recovery-horizon prediction is an interesting approach. Namely, using the predicted horizon to decide when to rely on recovered adapters versus the frozen backbone.

Critical weaknesses

Limited validation of the recovery-based gating mechanism: The ability to predict when historical adaptations can no longer be reliably recovered is one of the central contributions. The provided predicted-versus-measured recovery horizon analysis is useful, but they don't fully establish whether those predictions lead to good downstream decisions. We don't know how often the gate rejects useful adapters or admits adapters that perform worse than the frozen backbone. Reporting these errors, together with the resulting accuracy loss relative to an oracle serving policy, would help establish whether accurate horizon prediction translates into effective forgetting decisions.

Characterising the accuracy–memory trade-off: The fixed-budget evaluation is a strength of the paper. However, the main comparisons consider only a single memory cap per dataset for Palimpsest. Evaluating accuracy across a broader range of storage budgets would help establish whether the improvements are robust to the selected budget and more fully characterise the accuracy–memory trade-off.

Questions for authors

Recovery-based gating: The predicted-versus-measured recovery horizon results are encouraging, but how well do these predictions translate into downstream serving decisions? For example, examining how often the gate rejects recovered adapters that would still outperform the frozen backbone, or admits adapters that reduce accuracy. Could the resulting accuracy loss relative to an oracle gating policy be quantified?

Memory-budget sensitivity: The main fixed-budget comparisons evaluate PALIMPSEST at one memory cap per dataset. How sensitive are the results to the choice of budget? Would the performance advantage persist across a broader range of memory constraints, particularly when competing approaches are also optimised for each budget?

Generalisability: How dependent is the effectiveness of the proposed gating mechanism on the frozen backbone already performing reasonably well on earlier tasks? Would the approach remain effective in settings where previously learned task-specific adapters provide capabilities that the backbone does not possess?

3 comments

We thank the reviewer for the positive assessment and for suggestions that directly test our central claim.

W1 / Q1. Do horizon predictions translate into good serving decisions?

The gate serves a task of age aa from its recovered adapter if a≤T⋆(m,α)a \le T^\star(m,\alpha) and falls back to the frozen backbone otherwise. As the reviewer suggested, we compare it with a per-task oracle that serves whichever of the two is more accurate for each task. The oracle uses test labels, so it is an upper bound. Over 12 (m,α)(m,\alpha) cells × 4 task orderings on ImageNet-R (T=10T{=}10; 480 task instances):

policyfalse rejects (useful adapter dropped)false admits (harmful adapter served)
always serve recovered033.8%
always frozen61.2%0
T⋆T^\star-gate (ours)33.1%0.2%
oracle (per task, uses labels)00

The gate recovers 65% of the oracle's gain over always serving, and it almost never serves a harmful adapter (0.2% false admits). Its errors are conservative: one in three instances is an adapter past T⋆T^\star that would still have helped, spread over ages T⋆+1T^\star{+}1 to T⋆+5T^\star{+}5. A per-task quality signal could recover part of this regret. Extending the gate beyond a single age threshold and deciding which tasks to keep under the same budget is selective forgetting, is our future work, and we will discuss it in the paper.

W2 / Q2. Sensitivity to the memory budget.

We agree that a single cap per dataset leaves the trade-off under-characterised. We now sweep the budget on ImageNet-R (T=40T{=}40, B∈{2,5,10}B\in\{2,5,10\}M) with few baseline method:

methodB=2B{=}2MB=5B{=}5MB=10B{=}10M
PALIMPSEST (ours)69.1970.3970.79
EASE-21k62.5165.4164.33
InfLoRA56.4058.5559.95
SD-LoRA53.7854.5851.33

During the rebuttal time since time is limited, we dont have enough time to cover all baseline cap sweep, but we will finish the results and add to the appendix for the full paper later.

Q3. Does the gate depend on a strong frozen backbone?

By introducing the gate mechnism, we wish the model can know when to fall back to frozen backbone, rather than using the degraded adaptor, which can cause harm to the final results. Identifying which adapters carry capabilities the backbone lacks, and protecting them preferentially, is a different problem — deciding what to forget rather than when will be our follow-up work on selective forgetting.

We thank the reviewer again and are happy to provide further details.

Dear Reviewer,

Since there is only few hours left for the rebuttal, could you please read the rebuttal and provide a response? This is important for the review process.

AC

Thanks to the authors for the response and additional experiments. The gate-oracle comparison helps to show that the gate can avoid serving harmful adapters, though the gate seems quite conservative.

The additional memory budget experiments also strengthen the paper.

Some limitations remain though around the central contribution, such as the extent to which the proposed approach provides benefits over simpler storage strategies. I am keeping my original score.

Reviewer_yZ5TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper wants to solve the problem in continual learning that if every task needs to keep its own adapter, the required storage grows linearly with the number of trained tasks, which is very resource consuming. If storage is limited, forgetting must happen and accuracy will drop. So the authors propose PALIMPSEST, which uses a fixed memory, a dictionary, and keys to save the parameters that need to be stored. The decay in the memory makes forgetting ordered, from the oldest task first, and a horizon formula T* predicts which tasks’ recovered adapters can be trusted and which should fall back to the frozen model.

Critical strengths

The predictable forgetting part is reasonable: under limited storage, forgetting inevitably happens. The paper proposes a way to derive the horizon T* from compressed-sensing recovery and uses it as a gate in pre-trained continual learning, falling back to the frozen backbone beyond T*. After fitting the constants once per domain, it only needs m and α to know in advance which parts can be trusted.

Critical weaknesses

In my understanding, what really carries the adapter information is the dictionary, but it is not compressed. Only the x_t part is compressed, but x_t itself is very small, so is the compression applied to a part that is not the main one? In this way, the dictionary still grows together with the tasks, and the fixed size only holds for the memory z. If this is the case, the good results may come from keeping only the top-k singular components of each adapter (a rank truncation), not from the superposition into z that the paper focuses on, so the main contribution is not clearly supported by the experiments.

Questions for authors
  1. I would like to know the distribution of storage use among the dictionary, prototypes, and store. Is the dictionary the largest part, and what is its growth during training? Was the choice to use k=1 atom per layer for the OmniBench-1K T=200 entry because it would exceed 15M otherwise?
  2. If x_t is sparse enough, what is the reason for choosing to add decay and store it in z? If z is not the largest part of the storage cost, wouldn’t directly storing x_t be more reasonable? Then there would be no need for the intermediate transformation, and the adapter could be recovered more accurately.
2 comments

We thank the reviewer for the precise reading and the concrete questions. We added a full storage breakdown, a fixed-size dictionary that never exceeds the budget, and the ablation the reviewer asked for, which separates rank truncation from the store.

Q1. Storage breakdown, dictionary growth, and k=1k{=}1 at T=200T{=}200.

Here is a breakdown of the storage:

OmniBench-1Kdictionaryprototypesstore zztotal
T=50T{=}50, k=2k{=}23.69M0.77M0.015M4.47M
T=100T{=}100, k=2k{=}27.37M0.77M0.015M8.16M
T=200T{=}200, k=1k{=}1 (reported)7.37M0.77M0.015M8.16M
T=200T{=}200, k=2k{=}2, growing dictionary14.75M0.77M0.015M15.53M > 15M
T=200T{=}200, k=2k{=}2, fixed dictionary (new)2.58M0.77M0.015M3.36M

The decay factor decrease the weight for the early adaptors, then once the early adaptor fails to pass the gate T⋆T^\star, the model will use the frozen backbone, in that case our original plan was to delete the unreliable adaptor to help the dictionary under budget, so we will redraft the table 1 again later with the following eviction results so all the case will be at k=2k{=}2

OmniBench-1K, ALastA_{Last} (%)frozenno evictionwith evictiondictionary
T=50T{=}50, k=2k{=}252.6062.7062.773.69M → 2.58M
T=100T{=}100, k=2k{=}252.2762.4762.517.37M → 2.58M
T=200T{=}200, k=2k{=}252.3459.7162.3414.75M → 2.58M
T=200T{=}200, k=1k{=}152.3461.2762.297.37M → 1.29M

Weakness / Q2. Is the gain from rank truncation rather than superposition? Why not store xtx_t directly?

With per-task atoms, xtx_t is only 24k24k floats per task, so the saving relative to per-task adapters comes from the rank-kk atoms. Below are the comparison between store xtx_t and store zz:

ALastA_{Last} (%)OB1K T=50T{=}50OB1K T=100T{=}100OB1K T=200T{=}200 (k=1k{=}1)ImageNet-R T=40T{=}40
frozen backbone52.6052.2752.3454.17
rank-kk truncation, xtx_t stored exactly (no zz)62.5461.4362.2971.62
PALIMPSEST (zz, decay)62.7062.4761.2770.39
full rank-10 adapters62.5262.4661.3272.17

There is not a lot difference between store xtx_t or zz The accuracy comes from the rank-kk adapters. What zz provides is different: it makes the forgetting horizon computable in advance.

  • Without zz (storing xtx_t), recovery is exact and nothing is forgotten until an external rule deletes it. How long a task stays usable is then a free hyperparameter, with no model of when the budget runs out.
  • With zz, all tasks share one budget (m,α)(m,\alpha). Decay orders the forgetting (oldest first), and compressed-sensing recovery gives the age at which a task stops being recoverable, T⋆(m,α)T^\star(m,\alpha), in closed form before the stream runs. The gate, the certificate and the dictionary cap above all use it, and Sec. 4 validates it (forecast R2R^2 = 0.84–0.95).

We will revise the abstract and Sec. 3 accordingly: compression is attributed to rank-kk atoms, and zz is presented as the mechanism that makes forgetting predictable.

We hope this clarifies the contribution and would be glad to answer further questions.

Thank you for the detailed response and additional experiments. I appreciate that the authors addressed my original concerns, especially the storage breakdown and the direct comparison with storing xtx_t.

However, my main concern remains. The new experiments suggest that most of the accuracy improvement comes from rank-kk truncation rather than the superposition store zz. Directly storing xtx_t achieves comparable or even better accuracy.

The authors now emphasize predictable forgetting as the main contribution. However, I am still not convinced why superposition is necessary to achieve this goal. What practical advantage does predicting the recovery failure of zz provide over directly storing xtx_t under the same memory budget?

I think a matched-budget comparison with direct sparse-code storage and an explicit retention policy is necessary to demonstrate whether the proposed mechanism provides meaningful benefits beyond rank-kk compression.

Overall, while the rebuttal clarifies the authors' claims, my concerns about the necessity and significance of the proposed mechanism remain. I maintain my Weak Rejection.

Reviewer_5wdwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PALIMPSEST, a continual-learning method that consolidates each task's LoRA adapter into a fixed-size decayed store plus a growing pooled dictionary. The paper supports this design with a compressed-sensing reduction, a stable-recovery analysis, a support-recovery condition, and a calibrated horizon; experiments on ImageNet-R (T=10/20/40) and OmniBench-1K (T=50/100/125/200) show strong long-horizon accuracy with substantially lower retained storage than per-task adapter baselines.

Critical strengths
  1. Novel and elegant formulation. Casting continual learning as a single compressed-sensing problem (z=Aw⋆z = Aw^\star) is clean and impressive.
  2. The gate is a practical contribution. Using a calibrated horizon to decide which recovered adapters to serve, with fallback to the frozen backbone, is a sensible deployment policy. Although it may not fit the real world settings since the first learnt knowledge could be very important and less forgettable.
  3. Honest separation of proved vs calibrated. Appendix A.8 explicitly distinguishes the CS results from the calibrated recover to accuracy bridge and the load-collapse model.
Critical weaknesses
  1. The “memory-bounded” claim is overstated; the dictionary grows and dominates the state.
  2. The “shared dictionary” is not demonstrated to be shared. Each task appends its own top-(k) singular components; the code can in principle use atoms from earlier tasks, but the paper does not report how much it does.
  3. The sparse-coding step is underspecified, and the theory rests on unmeasured quantities. It seems that the paper does not give the solver? Or I missed somewhere?
  4. The age-only gate ignores per-task recovery quality. Using a global age-only horizon with domain-calibrated constants seems a bit wired to me.
  5. The experiments train all tasks, solve once, and then evaluate. In an online continual-learning deployment, the store changes after each task. So one would need (O(T)) solves over the stream, each with (N=Tp) growing? Can authors report the wall-clockiteration counts, or the number of solves over a stream?

The authors have solved most of my questions I have no more concerns on it.

Questions for authors

See the weaknesses

1 comment

We thank the reviewer for the careful reading, and we are glad you found the compressed-sensing formulation novel and elegant, the gate practical, and our separation of proved and calibrated results honest. We address each weakness below, with new experiments, and will revise the paper accordingly.

W1. "Memory-bounded" is overstated; the dictionary grows. The decay α\alpha bounds the store zz, not the dictionary, which is append-only. The decay factor here is to protect the new task and gradually decrease the weight for the old task, so with the gate mechnism once the model falls back to the frozen backbone, the old memory is not useful anymore, so we will delete them, so the dictionary will be bounded by eviction. Tasks older than T⋆(m,α)T^\star(m,\alpha) are no longer reliably recoverable; this is exactly where the gate falls back to the frozen backbone. We therefore evict them,here is a results what happens if we delet the non-reliable adaptor atoms, one thing to clarify here in case for confusion, the dictionary size under is pure dictionary storage size, and the 'store' showed inside the paper table is the overall storage size.

OmniBench-1K, ALastA_{Last} (%)frozenno evictionwith evictiondictionary
T=50T{=}50, k=2k{=}252.6062.7062.773.69M → 2.58M
T=100T{=}100, k=2k{=}252.2762.4762.517.37M → 2.58M
T=200T{=}200, k=2k{=}252.3459.7162.3414.75M → 2.58M
T=200T{=}200, k=1k{=}152.3461.2762.297.37M → 1.29M

From the table we can see, there is nearly no harm for deleting the adaptor's atom from the dictionary once they can not be trusted, in this way we can keep the dictionary size bounded.

W2. The "shared dictionary" is not shown to be shared. We agree and will correct the wording. In our coder each task's code is supported on that task's own top-kk singular atoms. The atoms are pooled in a common index space, so the joint ℓ1\ell_1 solve could place mass on earlier atoms, but the coder does not share them by construction. We will reframe it in the paper and remove any wording that implies measured cross-task reuse.

W3. The sparse-coding solver is underspecified. Thank you so much for pointing it out, We are sorry that we didn't mention the l1l_1 solver in the paper, the solver is FISTA (Beck & Teboulle, 2009): a fixed 300 iterations, step 1/L1/L with LL estimated by 20 power iterations on A⊤AA^\top A, λ=0.02 ∥A⊤z∥∞\lambda=0.02\,\|A^\top z\|_\infty, zero initialisation, no debiasing. Keys have i.i.d. N(0,1/m)\mathcal N(0,1/m) entries and are regenerated from integer seeds, never stored; the block of Φt\Phi_t acting on task ss's atoms has its own seed, so keys stay fixed as the dictionary grows and evicted blocks are simply dropped.

W4. The age-only gate ignores per-task recovery quality. The age-only gate is a deliberate choice: it needs no labels and no per-task validation data at deployment, and it is conservative. We agree that early-learned knowledge may deserve extra protection in some cases. But under a fixed budget something must be forgotten, and choosing which tasks to keep is selective forgetting, which we are developing in follow-up work. Our horizon answers the complementary question of when forgetting happens, so the budget can be provisioned in advance for the retention window a practitioner needs. We will add this to the Limitations.

W5. Online cost: O(T)O(T) solves with growing NN. We follow the standard PTM-CIL protocol (L2P, InfLoRA, SD-LoRA, …), which evaluates after the stream. To answer the deployment question, we timed one re-solve after every task on OmniBench-1K. One solve is 300 FISTA iterations, and a stream of TT tasks needs TT solves:

one solve at t=Tt{=}TTT solves over the streamwindowed solve (age ≤T⋆\le T^\star)TT windowed solves
T=50T{=}50, k=2k{=}23.8 min1.1 h1.9 min0.9 h
T=100T{=}100, k=2k{=}217 min10.7 h1.8 min2.4 h
T=200T{=}200, k=1k{=}147 min49 h0.9 min2.8 h

Since the current solver is just a CPU-algorithm based method, there will be other time-friendly solvers waiting to be tested out later if in some cases efficency is also a main concern

We hope these additions address the reviewer's concerns and would be happy to discuss further.

Comments on the paper

1 comment
Area_Chair_L2fPOpenReview ↗

Please engage in the discussion

Dear Reviewers,

As we enter the final stage of the author-reviewer discussion, I would like to encourage everyone to actively engage with the authors’ responses and any additional evidence they have provided.

Please take this opportunity to reconsider your initial assessments with an open mind, particularly where the authors have offered substantive clarifications, detailed arguments, or additional experimental results. The goal of the discussion is not to defend initial positions, but to reach the fairest and most informed evaluation of the submission.

I encourage you to engage directly with the authors on any remaining concerns, clearly distinguishing between issues that have been satisfactorily addressed and those that remain unresolved. Where appropriate, please update your reviews and scores accordingly, or explain why your original assessment still stands.

Your active and constructive participation in these final exchanges is essential to reaching a fair, evidence-based decision.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.