acceptodds
Under review as a conference paper at ICLR 2027

Nostalgia: Continual Learning through Constrained Optimisation in Small Models

11 views

Abstract

Pre-trained transformer models are typically adapted to downstream tasks via supervised fine-tuning, but sequential adaptation often leads to **catastrophic forgetting**, where performance on earlier tasks degrades. Existing mitigation strategies commonly rely on data replay, architectural modifications, or heuristic regularization, limiting their scalability. We introduce *Nostalgia*, a data-free, optimization-based method for continual supervised fine-tuning that mitigates forgetting by constraining parameter updates to directions that minimally interfere with previously learned tasks. Our approach is grounded in a local quadratic approximation of task losses and operationalizes this principle through a tractable low-rank approximation of the average Hessian, enabling efficient projection-based updates compatible with modern parameter-efficient fine-tuning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Life long learning, Continual learning, Optimisation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_QmvGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for training a model continually, using constrained optimization to prevent parameter updates from interfering with what was learned on past tasks. Instead of relying only on diagonal curvature, the method uses the Hessian matrix to capture interactions between parameters. A main contribution of the paper is an efficient way to maintain and update a rank-k memory of the average Hessian.

Critical strengths
  1. An efficient way to maintain and update a rank-k memory of the average Hessian. The Lanczos algorithm estimates each task’s dominant curvature directions from Hessian–vector products, and a factored update merges them into the memory at O(nk) cost.
Critical weaknesses
  1. The experimental evaluation is limited in both baselines and datasets. The ablation on the rank k is limited, and there is no ablation on the regularization coefficient λ. Reporting the actual training cost would also be informative, in particular the wall-clock time of the Lanczos estimation.
Questions for authors

None

1 comment

Thank you author for your invaluable feedback. We have taken your concerns into account and have updated the paper, with more robust experiments and have modified our claims accordingly. Thank you.

Reviewer_3sneOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses catastrophic forgetting in past-sample-free continual learning by maintaining a low-rank spectral approximation (Qt−1,Λt−1Q_{t-1}, \Lambda_{t-1}) of the running average Hessian across sequential tasks. Using Hessian-vector products (HVPs) with Lanczos tridiagonalization at task boundaries and Factor-SVD for cross-task accumulation, the method achieves O(nk)O(nk) storage and per-step computation for nn trainable parameters and rank kk, without storing replay examples. The authors propose two variants built on this shared curvature memory: Nostalgia, which applies hard gradient projection onto the orthogonal complement of the top-kk curvature subspace, and Nostalgia-EWC, which uses the low-rank spectral memory as a soft curvature-weighted quadratic anchor penalty. The authors also derive quadratic forgetting bounds under rank truncation and subspace estimation errors, and validate their methods on a 3-task ResNet-10 sequence (CIFAR-10 →\rightarrow MNIST →\rightarrow CIFAR-100) and a 6-domain subsampled DomainNet sequence using ViT-B/16 with LoRA.

Critical strengths
  1. Tractable Formulation of Off-Diagonal Curvature Memory for PEFT/Small Models: Classical regularization methods like EWC rely on a diagonal Fisher approximation that ignores cross-parameter interactions, while full Hessian storage requires prohibitive O(n2)O(n^2) memory. By combining matrix-free Lanczos HVPs with incremental Factor-SVD, the paper provides a clean, past-sample-free mechanism with O(nk)O(nk) complexity. This is consequential because it makes non-diagonal second-order curvature tracking computationally viable when paired with low-dimensional adapters such as LoRA.
  2. Controlled Comparison of Hard Subspace Projection vs. Soft Curvature Regularization: Because both Nostalgia and Nostalgia-EWC share the exact same low-rank spectral memory, the paper isolates the behavioral difference between instantaneous gradient projection and cumulative displacement anchoring. This directly demonstrates a practical stability-plasticity trade-off: hard projection preserves higher new-task plasticity, whereas soft quadratic anchoring yields substantially stronger retention on past tasks.
Critical weaknesses
  1. Severely Outdated Experimental Scale, Weak Baselines, and Single-Seed Reporting:

The empirical validation falls well short of current continual learning and continual fine-tuning standards. Evaluating a 3-task sequence (CIFAR-10 →\rightarrow MNIST →\rightarrow CIFAR-100) on ResNet-10 is essentially a toy setup that cannot test whether a fixed-rank (kk) Factor-SVD memory suffers from subspace crowding over longer task horizons (e.g., 10–20 tasks). Furthermore, despite positioning the work around pre-trained Transformer continual SFT and PEFT in the abstract and ViT experiments, the paper only compares against classical baselines from 2017–2021 (diagonal EWC, A-GEM, GPM). While the authors attempt to defend this scope in Section 9 by stating that newer methods (e.g., SDFT, SDPO) are limited to NLP/RL, this overlooks an extensive body of modern vision/CLIP continual fine-tuning baselines (such as ZSCL, WiSE-FT, L2-SP, and recent LoRA-based continual learning methods like O-LoRA or InfLoRA) evaluated on 10+ dataset sequences. Finally, the main benchmark tables rely on single-seed runs without variance or confidence intervals, making it impossible to verify the statistical significance of small performance margins.

  1. Disconnect Between Local Quadratic Null-Space Theory and Multi-Step Non-Convex Optimization:

The theoretical justification for Nostalgia relies on a local second-order Taylor expansion around an exact local minimum (∇Lτi(θτi∗)=0\nabla L_{\tau_{i}}(\theta^{\ast}_{\tau_{i}}) = 0) where parameter updates remain within a strictly quadratic neighborhood. In realistic multi-epoch continual fine-tuning, models rarely converge to exact zero-gradient minima (leaving non-zero first-order drift), and multi-step parameter trajectories may quickly exit the local region where the static Hessian H(θ∗)H(\theta^{\ast}) is valid. This theoretical fragility is confirmed by the paper's own empirical results: the theoretically motivated hard null-space projection (Nostalgia) suffers substantially higher forgetting than the heuristic distance anchor (Nostalgia-EWC), because instantaneous gradient projection fails to bound cumulative parameter displacement along a non-convex trajectory.

Questions for authors
  1. Empirical Robustness:

Given that the difference on ResNet-10 and DomainNet are close (~0.5%), can you provide multi-seed results (mean ±\pm std over at 3–5 seeds) to confirm that the differences between Nostalgia, GPM, and Nostalgia-EWC are statistically significant?

  1. Baseline Selection in Vision PEFT:

In Section 9, you state that newer continual learning methods are restricted to NLP/RL and lack vision equivalents. However, in the ViT-B/16 + LoRA setting, why were modern vision/PEFT continual learning baselines (e.g., L2-SP, WiSE-FT, O-LoRA, InfLoRA, or ZSCL) omitted in favor of classical scratch-training baselines from 2017–2021?

  1. Clarification / First-Order Gradient Residuals:

The zero-forgetting condition in Equation (6) assumes ∇Lτi(θτi∗)=0\nabla L_{\tau_i}(\theta^*_{\tau_i}) = 0 at the end of each task. When fine-tuning for a small number of epochs, the gradient norm at task termination is non-zero. How sensitive are the theoretical forgetting bounds and the Lanczos curvature estimates (which may encounter negative eigenvalues in non-convex regions) to non-zero first-order gradients and non-PSD Hessian spectra?

1 comment

Dear Reviewer, Thank you for your feedback. We apologize for the delayed response but hope this answers some of the questions you have posed to us. Note that we have updated the paper to include multi-seed runs and more baselines. We have modified the paper's claims accordingly, too.

A1: Updated the paper to have more robust results. A2: We did include more baselines in the updated papers. A3: We understand that some of the assumptions stated in Equation (6) aren't exactly true, while we agree that this is true, we view it from the point of view of hueristics and reasonable enough for the method to work. The noise associated with loss surfaces are true, especially consider batched processing. We have now included senstivity studies in the paper as well with regards to the eigen specra across different batches.

Reviewer_QceROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Nostalgia and Nostalgia-EWC, two past-sample-free optimization methods for continual supervised fine-tuning. Grounded in a local quadratic approximation of past task losses, the approach maintains a rank-kk spectral approximation of the running average Hessian at O(nk)O(nk) storage and update cost for nn trainable parameters. Dominant eigenpairs are extracted after each task via Lanczos iterations using Hessian-vector products and merged across tasks via factored SVD. Nostalgia projects gradients onto the orthogonal complement of the retained curvature basis, whereas Nostalgia-EWC applies a soft quadratic penalty weighted by the low-rank Hessian memory. The methods are evaluated on a three-task ResNet-10 sequence and a six-domain DomainNet sequence using ViT-B/16 with LoRA.

Critical strengths

Compatibility with Parameter-Efficient Fine-Tuning: Restricting second-order Hessian-vector products and spectral storage to LoRA adapter parameters makes curvature-constrained continual learning computationally practical for vision transformers.

Critical weaknesses
  • Limited Novelty Over Existing Low-Rank and Sketched Structural Regularization: The core idea of Nostalgia-EWC—replacing EWC's diagonal matrix with an O(nk)O(nk) low-rank or factored approximation of the n×nn \times n curvature matrix by exploiting fast spectral decay—has been extensively studied in the continual learning literature that is neither cited nor compared against. In particular, Li et al. (2021, Lifelong Learning with Sketched Structural Regularization) already formulated O(nk)O(nk) factored quadratic structural regularizers, explicitly evaluated rank-kk SVD EWC against diagonal and block-diagonal EWC, and showed that linear sketching (CountSketch) approximates the quadratic form in a single pass with simple linear online aggregation—avoiding iterative Lanczos double-backprop HVPs and repeated SVD merging. Other standard non-diagonal curvature methods such as Online Structured Laplace / K-FAC (Ritter et al., 2018) and Rotated EWC (Liu et al., 2018) are also omitted from both the related work and experimental baselines.

  • Single-Seed Runs and Missing Baselines: Almost all experimental results in Tables 1, 2, and 3 rely on a single random seed, and key baselines including Naive fine-tuning, standard EWC, and A-GEM are omitted from the main DomainNet comparison in Table 1. Explicitly deferring multi-seed runs and baseline comparisons to a future revision leaves the empirical claims unverified.

  • Contradictory Quantitative Tables: In Table 1, Nostalgia-EWC reports an average forgetting of 0.023 across the five forgettable DomainNet domains. However, in Table 2, the reported per-domain forgetting values for Nostalgia-EWC are 0.012, -0.039, 0.006, 0.529, and 0.002, which average to 0.102.

  • Ambiguous Implementation Details and Poor Manuscript Polish:

  1. Unclear multi-task architecture: For the ResNet-10 sequence across CIFAR-10, MNIST, and CIFAR-100, the paper does not explain how 1-channel 28×2828 \times 28 MNIST images and 3-channel 32×3232 \times 32 CIFAR images or the 10-way versus 100-way classifier heads are unified.

  2. Inconsistent projection description: Appendix A.1 states that Lanczos HVPs are computed over flattened backbone parameters while gradient projection is applied separately to each parameter group, even though sub-blocks of a globally orthonormal matrix Qt−1Q_{t-1} are not orthogonal projectors.

  3. Presentation flaws: The abstract frames the paper around pre-trained transformers while the introduction immediately states the work studies a small convolutional network on a three-task sequence; Algorithm 1 is titled Nostalgia but implements Nostalgia-EWC; line 346 contains a broken reference to Table 5.2.1.

Questions for authors
  1. Global Lanczos Basis versus Per-Group Projection: In Appendix A.1, how is gradient projection applied separately to each parameter group when the orthonormal basis Qt−1Q_{t-1} is computed jointly across flattened backbone parameters?
1 comment

We apologize for the delayed rebuttal, but we hope this helps. We have updated the paper. We thank you for highlighting some of the crucial drawbacks of the paper. About your keen observation on Appendix A.1, we made the following corrections: " one global basis Q over all flattened backbone params (utils/nostalgia.py:44-47,68-72,125-141); the concatenated gradient is projected once by I − QQᵀ (code: QQᵀ with scaling). No per-group projector exists.". Thank you for your feedback.

Reviewer_AizkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes two continual learning methods that retain a low-rank curvature matrix to identify parameter directions along which updates may substantially increase past-task losses. Both methods operate without storing examples from previous tasks. Nostalgia projects updates onto the orthogonal complement of the retained subspace. Nostalgia-EWC penalizes displacement within that subspace in proportion to the estimated curvature.

Critical strengths
  1. Lanczos estimation with Hessian–vector products and factored updates preserves cross-parameter curvature structure without retaining past examples. The rank-kk memory has O(nk)O(nk) storage and per-update cost and can be applied to LoRA parameters.

  2. Both variants use the same curvature-accumulation procedure. Nostalgia projects gradients onto the orthogonal complement of the retained basis, while Nostalgia-EWC penalizes squared displacement along each protected direction in proportion to its estimated curvature.

  3. Under the local quadratic assumptions of Eq. (18), Theorems 3 and 5 bound average past-task forgetting in terms of the first omitted eigenvalue σk+1\sigma_{k+1} and the error from imperfect subspace estimation ε2σ1\varepsilon^2\sigma_1.

Critical weaknesses
  1. Replacing EWC’s diagonal Fisher with a low-rank Hessian can also change the effective regularization strength. DomainNet omits standard EWC, and Figure 3 varies kk only for hard projection. The individual effects of these changes have not been sufficiently isolated.

  2. It seems that comparisons with the closest prior work are incomplete. For Nostalgia-EWC, CSQN-CT also accumulates curvature factors using SVD, and Kronecker-factored online Laplace retains cross-parameter curvature structure. Comparisons at similar storage cost would clarify the benefit of Lanczos estimation.

  3. The comparison between hard projection and soft regularization is confounded by differences in the optimizers used. While Section 5.1 states that a common AdamW optimizer was employed across all methods, Appendix A.2 explains that momentum and coordinate-wise preconditioning were omitted only for the hard-projection variant. Learning trajectories will differ depending on whether the entire AdamW update was projected, whether raw gradients were projected and directly applied, and how weight decay was handled.

  4. The connection between the forgetting bound and Nostalgia-EWC needs clarification. Theorems 3 and 5 concern hard projection under the local quadratic assumptions and zero prior forgetting. For the soft objective, Eq. (12) omits the linear term that generally remains at the most recent task solution.

Questions for authors
  1. At matched final-task accuracy, obtained by sweeping λ\lambda for both methods over multiple seeds, does Nostalgia-EWC still retain earlier tasks better than standard EWC? Does the gain persist when QΛQ⊤Q\Lambda Q^{\top} is replaced by its diagonal at matched trace?

  2. How does Nostalgia-EWC compare with CSQN-CT and Kronecker-factored online Laplace at similar storage cost? What does Lanczos estimation provide over these curvature estimators?

  3. What exact update rule does hard-projection Nostalgia use: is the raw gradient or the full AdamW update projected, and how is weight decay handled? Does hard projection still retain earlier tasks worse than Nostalgia-EWC when both use the same optimizer, for example AdamW with only the projection added?

  4. Which derivation justifies omitting the linear term in Eq. (12) when the anchor is the most recent solution? Can Theorems 3 and 5, or an analogous bound, be extended to Nostalgia-EWC?

1 comment

Thank you for your feedback. I understand that this reply is a little too late. We have updated our methods and baselines to now include a 3-seed round and a wider suite of experiments that we weren't able to carry out previously. With regard to the questions you have posed to us:

Q1 Answer. On ResNet-10, EWC Pareto-dominates N-EWC: lower forgetting (0.071 vs 0.102) and higher final accuracy (0.676 vs 0.633). On DomainNet, the two are already matched on final accuracy within noise (0.571 vs 0.570 ± 0.004/0.002). There, EWC retains better (−0.026 vs −0.011) but is less plastic (0.549 vs 0.562), and both beat naive. So the honest claim is a trade-off, not a win.

Q2 Answer. We haven't been able to carry out CSQN-CT due to the current time constraints. Regarding K-FAC, we have implemented Online Structured Laplace with tuning (3 seeds, as before). We can report that K-FAC's per-layer factorisation keeps it cheaper than our nk memory at this rank, yet it is the weakest non-adapter method on ResNet-10 (MNIST forgetting 0.653 vs EWC 0.141) and only ties naive on DomainNet.

Q3 Answer: The raw gradient is projected inside the optimizer.step(), before the base optimizer consumes it. The AdamW update is therefore applied to the already-projected gradient.

Q4 Answer: Dropping it is justified only for the most recent task: the anchor θ* = θ_{t−1} is that task's solution, so its gradient vanishes up to optimizer noise. For older tasks, the linear term is not zero, and we now say in the paper that it is dropped as an approximation, not that it vanishes.

We apologies for the delayed response. However we do also thank you for the invaluable feedback, to help us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.