acceptodds
Under review as a conference paper at ICLR 2027

PAI: Plasticity-Anchored Initialization for Improved Stability-Plasticity Trade-offs in Continual Learning

Abstract

Continual learning (CL) requires models to remain adaptable to nonstationary task streams while preserving previously acquired knowledge. Yet the standard practice of warm-starting each task from the preceding task solution can progressively impair plasticity: over long task sequences, we observe increasing spectral concentration and decreasing stable rank, accompanied by deteriorating adaptation to new tasks. Existing approaches often restore plasticity by resetting or modifying parameters, but such interventions risk disrupting learned representations and exacerbating forgetting. To address this challenge, we introduce Plasticity-Anchored Initialization (PAI), a simple task-boundary initialization strategy that exploits the learning trajectory itself to balance stability and plasticity. PAI maintains two complementary parameter states: a fast-memory anchor, given by the latest task solution, that preserves recent adaptation, and a slow-memory anchor, formed by averaging historical task endpoints, that aggregates information across the learning trajectory. Before each new task, PAI interpolates between these anchors, moderating task-specific drift while retaining learned structure and steering optimization toward a more plastic parameter region. Theoretically, we analyze how single-sample perturbations propagate across task boundaries and show that, under matched assumptions and learning rates, PAI admits a task-wise uniform-stability upper bound no larger than that of standard sequential SGD. Empirically, across class-incremental learning, all-data replay, and 5,000-task ImageNet sequences, PAI consistently improves both knowledge retention and late-stage adaptation across CNN and Transformer architectures. These results establish task-boundary initialization as a simple yet effective mechanism for sustaining plasticity in long-horizon continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_4cYKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents PAI, a novel method that balances the stability-plasticity tradeoff in continual learning. PAI achieves this by interpolating a fast memory anchor and a slow memory anchor, which are the latest checkpoint and the average of the previous task endpoints, respectively. The paper attributes the effectiveness of PAI to its ability to decrease spectral concentration, based on empirical observation. Theoretical analysis shows that PAI exhibits better algorithmic stability, which is connected to generalization, than vanilla SGD by comparing the upper bound of the generalization gap. PAI is evaluated across diverse continual learning scenarios, including those with limited and full data access. PAI outperforms previous continual learning (CL) baselines, including both stability- and plasticity-oriented methods.

Critical strengths

None

Critical weaknesses
  1. Limited novelty:
  • The idea of interpolating fast and slow learners is already presented by Hare and Tortoise (HnT) [1]. The paper does not explain key differences between PAI and HnT, nor identify limitations of HnT or how PAI solves it. Based on their similarity, I believe HnT should be a core baseline and included in all experiments.
  • The argument that spectral properties of neural networks is connected to the plasticity has been already proven by several existing works [2-5], and those works already provide strong empirical and theoretical evidence that spectral concentration deteriorates plasticity. Therefore, the claim on spectral concentration is not novel at all.
  1. Limited baselines & insufficient evidence for claims:
  • Since the paper attributes PAI's effectiveness from decreasing spectral concentration, the experiment should include related methods that mitigate spectral concentration as baselines [2, 3, 5].
  • The claim that PAI mitigates spectral concentration is only supported by empirical evidence in Figure 1. The paper does not provide explanation on why PAI improves spectral property of the network nor what is advantage of PAI compared to existing methods that mitigates spectral concentration [2-5].
  • The theoretical part does not relate to stability-plasticity tradeoff. The theory only explains generalization capability of PAI, and does not reveal how PAI handles stability-plasticity tradeoff. In addition, since the better stability comes from incorporating previous task endpoints, the existing methods that leverage similar ideas (e.g., HnT) should be included in the analysis to check whether PAI offers better bounds.

References

[1] Lee, Hojoon, et al. "Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks." arXiv preprint arXiv:2406.02596 (2024).

[2] Lewandowski, Alex, et al. "Learning continually by spectral regularization." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Chung, Wesley, et al. "Parseval regularization for continual reinforcement learning." Advances in Neural Information Processing Systems 37 (2024): 127937-127967.

[4] Han, Isaac, et al. "FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff." arXiv preprint arXiv:2602.08040 (2026).

[5] Rosseau, Andries, Robert Müller, and Ann Nowé. "Preserving Plasticity in Continual Learning via Dynamical Isometry." arXiv preprint arXiv:2606.09762 (2026).

Questions for authors
  1. What is the key difference or advantage of PAI compared to HnT?
  2. How does PAI mitigate spectral concentration?
  3. How is the theoretical analysis connected to the stability-plasticity tradeoff?
Reviewer_H9R2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PAI initializes each new task by interpolating between the latest task endpoint and an arithmetic average of historical endpoints. Within-task training stays unchanged. The paper connects this intervention to weight spectra and derives stability and generalization bounds. Experiments cover limited-replay class-incremental learning, all-data replay, and 5,000 binary ImageNet tasks, with improvements in accuracy, forgetting, and current-task adaptation.

Critical strengths
  1. Simple implementation. PAI changes only task-boundary initialization and requires one additional parameter state. It is easy to combine with existing training methods.

  2. Consistent gains. Adding PAI improves mean final accuracy in all 18 BFP/SARL/IDER configurations in Tables 1-2. Appendix C.2 reports lower mean forgetting in the corresponding configurations. The long-sequence and all-data-replay experiments provide additional evidence.

  3. Useful ablations. Tables 3-4 examine alternative initializations and anchoring weights. Appendix B.5 clearly states the limits of the theoretical results.

Critical weaknesses
  1. Limited novelty. Hare & Tortoise (Lee et al., ICML 2024) already reinitializes a fast learner from a slowly averaged model. PAI replaces this with arithmetic averaging of task endpoints and partial interpolation. I find this a small methodological change, and the paper does not establish a clear advantage over the underlying idea. Hare & Tortoise appears in Table 1 but is absent from the main plasticity comparisons in Figures 3-4. Table 4 also omits the EMA coefficient and its tuning procedure.

  2. Weak practical motivation for fixed capacity. The main class-incremental experiments use CIFAR and Tiny-ImageNet with fixed backbones and small replay buffers. They do not explain what application requires this combination of constraints. A limit on stored examples does not imply a limit on model parameters. PAI stores another full parameter copy, but gives no total-memory, deployment-size, or compute budget that justifies this allocation over additional trainable capacity. The all-data-replay experiment removes the data-retention constraint, while the 5,000-task experiment measures current-task adaptation rather than retention of all previous tasks. These experiments leave the practical resource trade-off unresolved.

  3. Insufficient explanation of the benefit over selective resetting. WSC, FIRE, and CBP already aim to restore adaptability while preserving useful knowledge. The paper includes these baselines, but does not explain when historical interpolation is a better choice than selectively refreshing unimportant parameters. Global interpolation can also alter useful parameters.

  4. Unclear mechanism and experimental details. The spectral evidence is correlational. The paper needs to specify the measured layers, aggregation procedure, and model state. Figure 4's second spectral point separates PAI from BP despite identical stated updates through task two; the measurement timing needs clarification. Optimizer-state handling, scheduler resets, batch-normalization buffers, and replay sampling are also important for reproducing the initialization benefit.

Questions for authors
  1. What does arithmetic endpoint averaging solve that existing historical-weight initialization does not? Please clarify EMA tuning and provide a matched Hare & Tortoise comparison in a main plasticity experiment.

  2. What practical resource budget motivates fixed capacity and the additional anchor copy? How does PAI compare with allocating that budget to more trainable parameters?

  3. Under what conditions does historical interpolation work better than selective resetting, and why?

  4. Please define the spectral measurements and clarify the early divergence in Figure 4, along with optimizer, scheduler, and batch-normalization state handling.

Reviewer_1P7wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets plasticity loss in continual learning, where each task usually starts from the previous task's endpoint. PAI instead starts each task from an interpolation between the latest endpoint and the uniform average of all past endpoints, with weight β on the average. The method is motivated by rising spectral concentration (SC@10) and falling stable rank under sequential training (Fig. 1). Thm. 4.7 shows that PAI's upper bound on the generalization gap is no larger than that of sequential SGD. Experiments cover class-incremental learning with limited replay, a 20-stage setting that replays all data, and a 5,000-task ImageNet sequence without replay. PAI is compared with continual-learning and plasticity-oriented baselines, and is also added to three replay-based methods.

Critical strengths
  1. The method is simple and cheap. It changes only the task-boundary initialization and stores one extra copy of the parameters, so any real gain would be easy to adopt.
  2. The empirical gains are consistent. PAI has the highest mean final average accuracy (FAA) in all six columns of Table 1, and added to BFP, SARL and IDER it raises mean FAA in all 18 cells of Table 2. If the choice of β holds up (Question 2), this would make a good case for the paper on its own.
Critical weaknesses
  1. The explanation of why PAI works is not convincing.
    • SC@10 does not track plasticity across methods. In Fig. 3(b)-(c), DASH ends with SC@10 near BP's yet reaches about 80% accuracy, against about 66% for BP. The quantity the paper diagnoses therefore does not follow the property it cares about (Question 5).
    • Thm. 4.7 does not carry the claim either. It bounds the generalization gap of a simplified PAI, not forgetting or plasticity (L1730-1740). Its bound is smallest at β = 1, where PAI performs at the level of plain replay, 44.56 against 44.46 (Table 3).
  2. The gains rest on a sensitive β, and neither its selection nor its transfer is shown.
    • No validation split is described, and the two PAI entries in Table 1 that Table 3 covers, 54.25 and 27.92, are the best values of the β sweep, both at β = 0.85.
    • Beating the strongest baseline of Table 1 needs β between 0.8 and 0.9 on both datasets. On CIFAR-10, FAA drops from 54.06 at β = 0.9 to 44.56 at β = 1.0, Table 3 reports no standard deviations, and the β used for Tiny-ImageNet, CIFAR-10 with buffer 500 and the 5,000-task runs is not reported.
    • Since the experiments carry the paper, this is the main reason for my rating (Questions 1 and 2).
Questions for authors
  1. [Clarification] How were β and the baselines' hyperparameters chosen, and on which data? Which β was used in each setting?
  2. [Requested analysis] Can you report results with one β chosen on held-out training data of one benchmark and reused unchanged on the others, and add standard deviations to Table 3? If PAI still beats the strongest Table 1 baseline, that would be reassuring.
  3. [Clarification] Is the lower SC@10 already present at the averaged initialization, or does it appear only during training on the new task? Are the classifier head and BatchNorm statistics averaged too? In the limited-replay runs, averaging the head might mainly reduce the bias toward recent classes rather than restore plasticity.
  4. [Clarification] In Fig. 4(d), PAI's SC@10 at task 2 is about 0.11 against about 0.37 for BP and the other baselines. Yet Algorithm 1 (c_2 = 1) and App. B.5 (L1744-1746) imply that PAI and BP coincide through task 2. When is SC@10 measured, and does the code match Algorithm 1?
  5. [Clarification] In Fig. 3(b)-(c), DASH ends with SC@10 near BP's, yet reaches about 80% accuracy against about 66% for BP. How is the spectral account meant to relate to this result?
Reviewer_GQ4NOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Plasticity-Anchored Initialization (PAI), a change made only at task boundaries in continual learning. Each new task starts from an interpolation between the latest task endpoint (they called it “fast anchor”) and the running arithmetic mean of all previous endpoints (“slow anchor”), weighted by a coefficient β\beta. Training within each task is unchanged. The motivation comes from a 5,000-task ImageNet experiment where they observed spectral concentration rises and stable rank falls in the weights while current-task accuracy degrades. On the theory side, the authors use an analysis of single-sample SGD to show that PAI’s bound on the generalization gap is never larger than that of sequential SGD. Experiments cover class-incremental learning with limited replay, all-data replay, and long task sequences. PAI is added on top of several knowledge preserving approaches to assess compatibility, with improvements in final accuracy and forgetting.

Critical strengths

1- The method is very simple, cheap (one extra copy of the parameters) and easy to add to existing methods. The gains over vanilla replay are large (for example, +9.8 points on CIFAR-10 with buffer 200).

2- The evaluation is relatively broad: three continual-learning regimes, four architectures including a ViT, and runs with 5,000 tasks. PAI also improves forgetting when added to rehearsal-based methods.

3- The paper is well-written, clear, and easy to follow and understand, except for Section 4 (please refer to the Weaknesses section).

Critical weaknesses

1- My major concern is the novelty of the work, which I elaborate on below.

(a): Rank degradation as a signature of plasticity loss is well documented already, both in representations and in weight spectra [1, 2, 3] (and [4] which is not cited but directly experimented on spectral rank of weights). More recently in the last ICLR, [5] track effective and stable rank under continual learning. They also formalize a “rank-plasticity tension” that closely matches the mechanistic hypothesis stated here. This paper is not cited. Given all of these, the first listed contribution (“we identify increasing spectral concentration…”) overstates what is new.

(b): The method is very close to existing fast/slow weight schemes. The “EMA initialization” variant in Table 4 is essentially a Hare & Tortoise-style [6] reset toward a slow average. PAI differs mainly in using an arithmetic rather than an exponential average, and in having access to task boundaries, which allows it to perform averaging at the end of each task rather than periodically, as done in [6].

2- It seems to me that the theoretical results are only loosely connected to the practical algorithm. I elaborate on this below.

(a): Overall, I found the theoretical results (section 4) dense and incoherent. No intuition is provided for the assumptions, and the theorems are not clearly connected to one another.

(b): Based on my understanding, the key assumption in the theoretical result is a single convex terminal region shared by all β\beta, with bounded loss and gradients. It is strong for deep networks and never checked empirically.

(c): The provided bound is monotonically minimized at $\beta =1 $(Lemma B.8), yet β=1\beta=1 is one of the worst settings in practice (44.56% on CIFAR-10, Table 3). So the theory does not explain why an intermediate β\beta works.

(d): The bound concerns the generalization gap, not forgetting or plasticity, which are the quantities the paper is about. The section heading “Why historical anchoring helps” claims more than the analysis shows.

(e): Also, based on my understanding, the theoretical results show that the generalization gap is no worse than that of SGD, but they do not provide any guarantee of improvement, even under simplified assumptions.

3- The spectral argument is undermined by the paper’s own Figure 3. On ViT, S&P and L2 have low spectral concentration but poor accuracy, while FIRE has high concentration and good accuracy.

4- The baselines change across experiments. Table 1 includes only one plasticity-oriented baseline (FIRE), yet the text claims PAI outperforms plasticity methods. The long-sequence and all-data-replay figures omit Hare & Tortoise and WSC, the closest competitors.

5- Hyperparameter search spaces are missing for PAI and for several baselines.

[1] S. Dohare, et al. Loss of Plasticity in Deep Continual Learning, Nature

[2] A. Lewandowski, et al. Learning Continually by Spectral Regularization. arXiv:2406.06811, 2024

[3] C. Lyle Understanding Plasticity in Neural Networks, ICML 2023

[4] SRR. Rohani, et al. NeuMoSync: End-to-End Neuromodulatory Control for Plasticity and Adaptability in Continual Learning, TMLR 2026

[5] A. Joudaki, et al. Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity, ICLR 2026

[6] H. Lee, et al. Slow and Steady Wins the Race Maintaining Plasticity with Hare and Tortoise Networks, ICML 2024

Questions for authors

1- Please position the work against [5] and the rank-collapse literature, and reframe the spectral observation as confirming known behavior rather than as a contribution. The contributions should be substantially sharpened.

2- To separate “learned history” from simple shrinkage, could you add an experiment that interpolates toward the initial weights W0W_0 or a frozen early checkpoint, and report weight-norm trajectories? With β=0.85\beta=0.85, the latest endpoint gets only about 15% weight late in the sequence, so the effect may be closer to S&P or L2-to-init than the framing suggests.

3- How does PAI differ from Hare & Tortoise in a way that matters? A head-to-head comparison in the long-sequence and all-data-replay settings would help. Could PAI work without task boundaries, for example with periodic anchoring?

4- How should the theory be read given that it favors β=1\beta=1 while experiments favor about 0.85? Is there a way to capture the plasticity or fitting side of the trade-off? Can you validate the terminal-region assumption, or report the bound’s numerical value on a small problem?

5- How are AdamW optimizer states and BatchNorm statistics handled when parameters are interpolated?

6- The spectral metrics are computed on weights, while the narrative is about representations. Adding activation effective rank and dead- or duplicate-unit fractions would make the spectral argument more convincing.

7- Why not consider other continual learning regimes, such as domain-incremental learning, concept drift, or memorization (as in [4])? Is the proposed model applicable to these settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.