- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies continual learning in a fixed memory constraint setting, focusing on how to retain useful knowledge as new tasks are learned and memory capacity becomes exhausted. The authors propose Palimsest, which compresses task-specific LoRA adapters into a shared, fixed-size memory and approximately reconstructs them when needed. The main idea is to make forgetting predictable by estimating how far back historical adaptations can be reliably recovered, and using that estimate to decide when to fall back to the frozen backbone model. Experiments on image classification benchmarks demonstrate improved accuracy over competing approaches under the evaluated memory constraints, alongside evidence supporting the proposed recovery-horizon predictions.
Critical strengths
-
Interesting and well-motivated problem: The paper addresses continual learning under fixed memory constraints, focusing on what happens when memory becomes saturated. I like the perspective that, rather than attempting to eliminate forgetting, the key idea is that the system should know approximately when it can no longer reliably recover an older adaptation, making forgetting predictable and manageable.
-
Integrating recovery and forecasting: Their combination of sparse coding, shared memory, and recovery-horizon prediction is an interesting approach. Namely, using the predicted horizon to decide when to rely on recovered adapters versus the frozen backbone.
Critical weaknesses
Limited validation of the recovery-based gating mechanism: The ability to predict when historical adaptations can no longer be reliably recovered is one of the central contributions. The provided predicted-versus-measured recovery horizon analysis is useful, but they don't fully establish whether those predictions lead to good downstream decisions. We don't know how often the gate rejects useful adapters or admits adapters that perform worse than the frozen backbone. Reporting these errors, together with the resulting accuracy loss relative to an oracle serving policy, would help establish whether accurate horizon prediction translates into effective forgetting decisions.
Characterising the accuracy–memory trade-off: The fixed-budget evaluation is a strength of the paper. However, the main comparisons consider only a single memory cap per dataset for Palimpsest. Evaluating accuracy across a broader range of storage budgets would help establish whether the improvements are robust to the selected budget and more fully characterise the accuracy–memory trade-off.
Questions for authors
Recovery-based gating: The predicted-versus-measured recovery horizon results are encouraging, but how well do these predictions translate into downstream serving decisions? For example, examining how often the gate rejects recovered adapters that would still outperform the frozen backbone, or admits adapters that reduce accuracy. Could the resulting accuracy loss relative to an oracle gating policy be quantified?
Memory-budget sensitivity: The main fixed-budget comparisons evaluate PALIMPSEST at one memory cap per dataset. How sensitive are the results to the choice of budget? Would the performance advantage persist across a broader range of memory constraints, particularly when competing approaches are also optimised for each budget?
Generalisability: How dependent is the effectiveness of the proposed gating mechanism on the frozen backbone already performing reasonably well on earlier tasks? Would the approach remain effective in settings where previously learned task-specific adapters provide capabilities that the backbone does not possess?
3 comments
Dear Reviewer,
Since there is only few hours left for the rebuttal, could you please read the rebuttal and provide a response? This is important for the review process.
AC
Thanks to the authors for the response and additional experiments. The gate-oracle comparison helps to show that the gate can avoid serving harmful adapters, though the gate seems quite conservative.
The additional memory budget experiments also strengthen the paper.
Some limitations remain though around the central contribution, such as the extent to which the proposed approach provides benefits over simpler storage strategies. I am keeping my original score.
We thank the reviewer for the positive assessment and for suggestions that directly test our central claim.
W1 / Q1. Do horizon predictions translate into good serving decisions?
The gate serves a task of age a from its recovered adapter if a≤T⋆(m,α) and falls back to the frozen backbone otherwise. As the reviewer suggested, we compare it with a per-task oracle that serves whichever of the two is more accurate for each task. The oracle uses test labels, so it is an upper bound. Over 12 (m,α) cells × 4 task orderings on ImageNet-R (T=10; 480 task instances):
The gate recovers 65% of the oracle's gain over always serving, and it almost never serves a harmful adapter (0.2% false admits). Its errors are conservative: one in three instances is an adapter past T⋆ that would still have helped, spread over ages T⋆+1 to T⋆+5. A per-task quality signal could recover part of this regret. Extending the gate beyond a single age threshold and deciding which tasks to keep under the same budget is selective forgetting, is our future work, and we will discuss it in the paper.
W2 / Q2. Sensitivity to the memory budget.
We agree that a single cap per dataset leaves the trade-off under-characterised. We now sweep the budget on ImageNet-R (T=40, B∈{2,5,10}M) with few baseline method:
During the rebuttal time since time is limited, we dont have enough time to cover all baseline cap sweep, but we will finish the results and add to the appendix for the full paper later.
Q3. Does the gate depend on a strong frozen backbone?
By introducing the gate mechnism, we wish the model can know when to fall back to frozen backbone, rather than using the degraded adaptor, which can cause harm to the final results. Identifying which adapters carry capabilities the backbone lacks, and protecting them preferentially, is a different problem — deciding what to forget rather than when will be our follow-up work on selective forgetting.
We thank the reviewer again and are happy to provide further details.