- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents PAI, a novel method that balances the stability-plasticity tradeoff in continual learning. PAI achieves this by interpolating a fast memory anchor and a slow memory anchor, which are the latest checkpoint and the average of the previous task endpoints, respectively. The paper attributes the effectiveness of PAI to its ability to decrease spectral concentration, based on empirical observation. Theoretical analysis shows that PAI exhibits better algorithmic stability, which is connected to generalization, than vanilla SGD by comparing the upper bound of the generalization gap. PAI is evaluated across diverse continual learning scenarios, including those with limited and full data access. PAI outperforms previous continual learning (CL) baselines, including both stability- and plasticity-oriented methods.
Critical strengths
None
Critical weaknesses
- Limited novelty:
- The idea of interpolating fast and slow learners is already presented by Hare and Tortoise (HnT) [1]. The paper does not explain key differences between PAI and HnT, nor identify limitations of HnT or how PAI solves it. Based on their similarity, I believe HnT should be a core baseline and included in all experiments.
- The argument that spectral properties of neural networks is connected to the plasticity has been already proven by several existing works [2-5], and those works already provide strong empirical and theoretical evidence that spectral concentration deteriorates plasticity. Therefore, the claim on spectral concentration is not novel at all.
- Limited baselines & insufficient evidence for claims:
- Since the paper attributes PAI's effectiveness from decreasing spectral concentration, the experiment should include related methods that mitigate spectral concentration as baselines [2, 3, 5].
- The claim that PAI mitigates spectral concentration is only supported by empirical evidence in Figure 1. The paper does not provide explanation on why PAI improves spectral property of the network nor what is advantage of PAI compared to existing methods that mitigates spectral concentration [2-5].
- The theoretical part does not relate to stability-plasticity tradeoff. The theory only explains generalization capability of PAI, and does not reveal how PAI handles stability-plasticity tradeoff. In addition, since the better stability comes from incorporating previous task endpoints, the existing methods that leverage similar ideas (e.g., HnT) should be included in the analysis to check whether PAI offers better bounds.
References
[1] Lee, Hojoon, et al. "Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks." arXiv preprint arXiv:2406.02596 (2024).
[2] Lewandowski, Alex, et al. "Learning continually by spectral regularization." International Conference on Learning Representations. Vol. 2025. 2025.
[3] Chung, Wesley, et al. "Parseval regularization for continual reinforcement learning." Advances in Neural Information Processing Systems 37 (2024): 127937-127967.
[4] Han, Isaac, et al. "FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff." arXiv preprint arXiv:2602.08040 (2026).
[5] Rosseau, Andries, Robert Müller, and Ann Nowé. "Preserving Plasticity in Continual Learning via Dynamical Isometry." arXiv preprint arXiv:2606.09762 (2026).
Questions for authors
- What is the key difference or advantage of PAI compared to HnT?
- How does PAI mitigate spectral concentration?
- How is the theoretical analysis connected to the stability-plasticity tradeoff?