- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether CL methods with subspace-based directional constraints benefit from a simple mechanism of spectral-capacity control. The authors observe that representative methods (GPM & LoDA) produce updates with smaller magnitudes and more concentrated spectra than sequential fine-tuning. Replaying their singular-value spectra while allowing update directions to be learned freely recovers comparable performance. Motivated by this observation, the paper introduces SPAR, a simple algorithm that uses brief probing updates to estimate task demand and adaptively allocate LoRA ranks for each task. Experiments on multiple CIL benchmarks demonstrate its effectiveness.
Critical strengths
-
The perspective of this work is novel and could provide conceptual value. Separating update directions from singular-value spectra reframes the discussion from whether task subspaces should be isolated or shared to whether these methods benefit from common capacity-control effects.
-
Replacing the original directional mechanisms with spectral constraints and recovering similar performance demonstrates that specific directional constraints are not the only route to enhance CL performances. This is an interesting empirical insight.
-
The methodology section is easy to follow. Comprehensive experiments on CIL and DIL benchmarks demonstrate the effectiveness and generality of SPAR. This work alse analyzed the additional training time of probe, leaving no further concerns on extra computational cost.
Critical weaknesses
The spectrum-replay analysis does not disentangle the effects of spectral shape and update magnitude, leaving the specific benefit of spectral control unclear.
The rank-reduction analysis does not isolate update dimensionality from update magnitude, limiting the explanation of reduced forgetting.
The cumulative demand-based allocation may overconstrain later tasks in long streams, without reassessing the relevance of historical demand under sustained distribution shifts.
Full-matrix probing raises memory-efficiency concerns, and its necessity relative to lightweight LoRA-based probing remains unclear.
Questions for authors
-
It remains unclear which aspect of spectrum replay is responsible for recovering performance. The observed recovery therefore does not establish whether the reference spectrum’s detailed shape is important or whether simply constraining update magnitude would suffice. A norm-only control that matches the reference Frobenius norm while preserving the learned relative singular values would clarify what spectrum replay reveals beyond ordinary update-magnitude regularization.
-
The connection between rank reduction and reduced forgetting remains insufficiently established. A smaller LoRA rank limits the number of independent update directions but does not constrain update magnitude. It remains unclear whether the retention gains arise from lower dimensionality or smaller update magnitude.
-
Accumulated historical scores can make adaptation increasingly restrictive in long task streams. When probe scores remain bounded, the current demand share decreases even when later tasks require substantial adaptation. A sustained distribution shift may therefore receive insufficient capacity. The algorithm lacks a mechanism to reconsider whether that historical demand still justifies restricting current adaptation.
-
Could the probe use a small LoRA adapter instead of full-matrix fine-tuning to reduce its memory requirements?
-
Could the authors briefly extend their discussion or analysis to importance-based regularization methods, such as EWC [1] and EKPC [2], and expert-based routing or aggregation methods, such as CKAA [3] and StPR [4]? In particular, could the spectral-capacity perspective help explain how these methods balance stability and plasticity, or suggest complementary improvements? Such an extension would clarify the scope of the findings and offer useful design insights beyond subspace-based directional constraints.
[1] Overcoming catastrophic forgetting in neural networks.
[2] EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning. International Journal of Computer Vision.
[3] CKAA: Cross-Subspace Knowledge Alignment and Aggregation for Robust Continual Learning.
[4] StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning.
I greatly appreciate the authors' response. In their rebuttal, the authors state that ''These three methods adapt layer- or module-wise ranks to improve the trade-off between downstream fine-tuning performance and parameter efficiency. SPAR addresses the stability–plasticity trade-off in continual learning.'' However, from my perspective, AdaLoRA, SoRA, and AutoLoRA also attempt to tackle the stability–plasticity trade-off in continual learning. Therefore, my concern regarding the novelty of this paper has not been adequately addressed.
In terms of experimental results, the proposed method does not outperform LoDA and E2-LoRA (two ICML 2026 works) by a significant margin in both replay-free and replay-enabled continual learning settings. I consider the results only competitive. As such, my doubts about the effectiveness of the proposed approach remain unresolved.
Although the paper is titled “RETHINKING” and the method section contains several interesting analyses, I did not find any surprising experimental insights or conclusions (from my viewpoint). This is another reason why I view the novelty as limited.
While the paper is well-written with rigorous reasoning, given my concerns about novelty and empirical effectiveness, I will keep my original rating.