acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Directional Constraints in Continual Learning through Spectral Capacity

Abstract

Subspace-based continual learning methods constrain update directions in seemingly contradictory ways: some isolate task updates in orthogonal subspaces to reduce interference, whereas others encourage subspace sharing to promote transfer. Despite these opposing designs, both families exhibit similar update-spectrum patterns that differ markedly from unconstrained fine-tuning. To isolate this spectral effect, we replace each method's directional constraint with a training-time constraint that matches the spectrum recorded from its updates while leaving update directions unconstrained. This spectrum-only intervention matches or even surpasses the original direction-constrained methods. We further find that precise matching of the full spectrum is unnecessary: controlling only update spectral capacity—measured by spectral energy and effective rank—through simple schedules of training factors such as LoRA rank already substantially improves continual learning. Building on this insight, we propose SPAR (Spectral Probing for Adaptive Rank), which uses a few probing updates to estimate each task's demand for update spectral capacity and allocates its LoRA rank accordingly. Across diverse benchmarks, this simple scheduler achieves competitive or superior performance without elaborate directional constraints or auxiliary mechanisms.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_id3gOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether CL methods with subspace-based directional constraints benefit from a simple mechanism of spectral-capacity control. The authors observe that representative methods (GPM & LoDA) produce updates with smaller magnitudes and more concentrated spectra than sequential fine-tuning. Replaying their singular-value spectra while allowing update directions to be learned freely recovers comparable performance. Motivated by this observation, the paper introduces SPAR, a simple algorithm that uses brief probing updates to estimate task demand and adaptively allocate LoRA ranks for each task. Experiments on multiple CIL benchmarks demonstrate its effectiveness.

Critical strengths
  1. The perspective of this work is novel and could provide conceptual value. Separating update directions from singular-value spectra reframes the discussion from whether task subspaces should be isolated or shared to whether these methods benefit from common capacity-control effects.

  2. Replacing the original directional mechanisms with spectral constraints and recovering similar performance demonstrates that specific directional constraints are not the only route to enhance CL performances. This is an interesting empirical insight.

  3. The methodology section is easy to follow. Comprehensive experiments on CIL and DIL benchmarks demonstrate the effectiveness and generality of SPAR. This work alse analyzed the additional training time of probe, leaving no further concerns on extra computational cost.

Critical weaknesses

The spectrum-replay analysis does not disentangle the effects of spectral shape and update magnitude, leaving the specific benefit of spectral control unclear.

The rank-reduction analysis does not isolate update dimensionality from update magnitude, limiting the explanation of reduced forgetting.

The cumulative demand-based allocation may overconstrain later tasks in long streams, without reassessing the relevance of historical demand under sustained distribution shifts.

Full-matrix probing raises memory-efficiency concerns, and its necessity relative to lightweight LoRA-based probing remains unclear.

Questions for authors
  1. It remains unclear which aspect of spectrum replay is responsible for recovering performance. The observed recovery therefore does not establish whether the reference spectrum’s detailed shape is important or whether simply constraining update magnitude would suffice. A norm-only control that matches the reference Frobenius norm while preserving the learned relative singular values would clarify what spectrum replay reveals beyond ordinary update-magnitude regularization.

  2. The connection between rank reduction and reduced forgetting remains insufficiently established. A smaller LoRA rank limits the number of independent update directions but does not constrain update magnitude. It remains unclear whether the retention gains arise from lower dimensionality or smaller update magnitude.

  3. Accumulated historical scores can make adaptation increasingly restrictive in long task streams. When probe scores remain bounded, the current demand share decreases even when later tasks require substantial adaptation. A sustained distribution shift may therefore receive insufficient capacity. The algorithm lacks a mechanism to reconsider whether that historical demand still justifies restricting current adaptation.

  4. Could the probe use a small LoRA adapter instead of full-matrix fine-tuning to reduce its memory requirements?

  5. Could the authors briefly extend their discussion or analysis to importance-based regularization methods, such as EWC [1] and EKPC [2], and expert-based routing or aggregation methods, such as CKAA [3] and StPR [4]? In particular, could the spectral-capacity perspective help explain how these methods balance stability and plasticity, or suggest complementary improvements? Such an extension would clarify the scope of the findings and offer useful design insights beyond subspace-based directional constraints.

[1] Overcoming catastrophic forgetting in neural networks.

[2] EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning. International Journal of Computer Vision.

[3] CKAA: Cross-Subspace Knowledge Alignment and Aggregation for Robust Continual Learning.

[4] StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning.

Reviewer_z71ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Subspace methods for continual learning either isolate tasks in orthogonal subspaces or make them share directions, and it is unclear why both work. The paper argues that the update spectrum, not the direction, is what matters. It shows that matching only the singular values of an unconstrained run to those of a subspace method recovers its accuracy. It then proposes SPAR, which probes each task with a short fine-tuning run and sets the LoRA rank from the probe's spectral capacity. SPAR matches recent methods on class-incremental benchmarks, improves on domain-incremental ones, and trains faster.

Critical strengths
  1. The paper is very well written.
  2. The spectrum replay experiment is the highlight, and the insight that the directions can be dropped and the accuracy comes back is interesting.
  3. The insight is actionable. Replacing stored subspaces and projections with a schedule on learning rate or rank is a real simplification, and SPAR delivers it with one scalar of state per layer and 2 to 3 times less training time than the strongest baselines.
  4. The evaluation is broad. It spans class-incremental, domain-incremental, imbalanced, and NLP settings, and adds a rank-search oracle and sensitivity studies that make the evaluation robust.
Critical weaknesses
  1. The paper claims that demand-aware rank allocation helps, but this is never tested against the obvious alternative. No table compares SPAR to a fixed-decay LoRA.

  2. Novelty is not positioned against the closest work. The separation of spectrum from direction is presented as the key step, but two prior papers already make it. SD-LoRA (arXiv 2501.13198), cited here only as a sharing method, writes each task update as a learnable magnitude times a normalized direction and reports that later tasks' magnitudes decline. EBLoRA (arXiv 2602.00722) writes each update as a scalar times orthonormal bases, which fixes a flat spectrum, and tests it by replacing singular values while keeping singular vectors, the same intervention as the spectrum replay here. Please provide a clear distinction from these methods.

  3. ImageNet-A is standard in this line of work, used by InfLoRA, SD-LoRA, and LoDA, and it is where accuracies are lowest and gaps between methods widest. I recommend the authors include this benchmark.

Questions for authors
  1. How does SPAR compare to a fixed 1/t rank decay with no probe on the main benchmarks and on the imbalanced sequences?
  2. Given that SD-LoRA and EBLoRA already decouple magnitude from direction and EBLoRA already replaces singular values while keeping singular vectors, what exactly does the work show beyond them? The related work section is short and does not adequately cite all relevant work.
  3. Can the authors report SPAR on ImageNet-A with 10 and 20 sessions, with and without classifier alignment, against the same baselines as Table 2?
Reviewer_UNquOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The spectrum-replay intervention (Tables 1 and 6) deliberately separates direction from spectral capacity: keeping current singular vectors but restoring the recorded singular values recovers or exceeds original performance for GPM, LoDA, SLCA, and BiLoRA. This empirical evidence supports the observation that the magnitude and effective-dimensionality profile of updates, rather than prescribed orthogonal or shared directions, is a useful control lens for continual learning, and it may prompt the community to revisit how directional constraints are theoretically justified.

Critical strengths

The paper studies why two seemingly opposed subspace-based continual learning families—separation-oriented methods that isolate updates in orthogonal subspaces and sharing-oriented methods that optimize shared directions—reach similar performance, and whether the prescribed directional constraint carries an independent effect. The authors remove directional constraints and instead match the recorded update spectrum (spectrum replay), showing that restoring spectral capacity while leaving singular vectors unconstrained matches or surpasses the original methods. They then find that precise full-spectrum matching is unnecessary: spectral capacity, measured by spectral energy and effective rank, can be controlled through simple decay schedules of learning rate, optimization steps, or LoRA rank. SPAR (Spectral Probing for Adaptive Rank) uses a brief probe epoch to estimate task demand from the probe spectrum and allocates LoRA rank accordingly.

Critical weaknesses
  1. Limited novelty against existing adaptive-rank LoRA methods. Mechanically, SPAR is a demand-aware LoRA rank allocator: one probe epoch estimates per-layer spectral energy and effective rank, and rank is scaled by their product st=Ft⋅reff,ts_t = \mathcal{F}_t \cdot r_{\mathrm{eff},t}. This setting overlaps substantially with existing adaptive-rank LoRA methods—AdaLoRA allocates rank by importance after SVD, SoRA selects ranks from gradient sparsity, and AutoLoRA learns rank selection over tasks. Section A.4 itself states SPAR targets the same objective as these methods, with the only stated difference being the heuristic demand score derived from probe spectra. The theoretical support in Appendix B.1 gives sufficient conditions only under homogeneous interference, and the authors explicitly note it does not imply a 1/t decay or an optimal scheduler. The central claim that spectral capacity control replaces directional constraints therefore rests on heuristic scheduling experiments rather than a non-trivial new mechanism.
  2. Modest experimental gains, with attribution not cleanly separated from classifier alignment. In the main CIL table, SPAR+CA improves over the strongest baselines by roughly one Last-Acc point or less on most benchmarks; on CIFAR-100 its Last-Acc 90.05 is actually below LoDA's 90.47. Table 5 shows the demand-score combination Ft⋅reff,t\mathcal{F}_t \cdot r_{\mathrm{eff},t} improves Last-Acc by only 0.57–1.42 points over equal-weight or inverse-rank scores, and the inverse score is essentially tied on the 9:1 setting. Because SPAR is evaluated with classifier alignment (CA) throughout, the reader cannot tell how much of the reported gain comes from the rank scheduler itself versus the external CA component; a clean ablation of SPAR without CA against LoDA, E²-LoRA, and the adaptive-rank baselines is missing.
Questions for authors
  1. Tables 2 and 3 report SPAR+CA throughout. Could the authors provide a clean comparison of SPAR without CA against LoDA, E²-LoRA, AdaLoRA, SoRA, and AutoLoRA, isolating the contribution of the rank-allocation policy from classifier alignment?
  2. Why is the demand score the product Ft⋅reff,t\mathcal{F}_t \cdot r_{\mathrm{eff},t} rather than a weighted sum, max, or another combination? Appendix B.2 acknowledges this is a heuristic without an optimality derivation; could the authors provide either a theoretical justification or a more thorough empirical comparison against alternative score forms beyond Table 5?
4 comments

I greatly appreciate the authors' response. In their rebuttal, the authors state that ''These three methods adapt layer- or module-wise ranks to improve the trade-off between downstream fine-tuning performance and parameter efficiency. SPAR addresses the stability–plasticity trade-off in continual learning.'' However, from my perspective, AdaLoRA, SoRA, and AutoLoRA also attempt to tackle the stability–plasticity trade-off in continual learning. Therefore, my concern regarding the novelty of this paper has not been adequately addressed.

In terms of experimental results, the proposed method does not outperform LoDA and E2-LoRA (two ICML 2026 works) by a significant margin in both replay-free and replay-enabled continual learning settings. I consider the results only competitive. As such, my doubts about the effectiveness of the proposed approach remain unresolved.

Although the paper is titled “RETHINKING” and the method section contains several interesting analyses, I did not find any surprising experimental insights or conclusions (from my viewpoint). This is another reason why I view the novelty as limited.

While the paper is well-written with rigorous reasoning, given my concerns about novelty and empirical effectiveness, I will keep my original rating.

We sincerely thank the reviewer for the further feedback. We would like to emphasize one fundamental distinction that is central to our contribution.

Our contribution is not another adaptive-rank method, but a different understanding of why directional constraints work in continual learning. While existing methods focus on prescribing update directions, our controlled experiments reveal that much of their benefit can be recovered simply by preserving the induced update spectra, without enforcing the original directional constraints (Table 1). This shifts the perspective from constraining where to update to controlling how much spectral capacity to use. SPAR is a simple and efficient realization of this insight, rather than merely another rank-allocation strategy.

Regarding AdaLoRA, SoRA, and AutoLoRA, although they can also be applied to continual learning, their original designs do not explicitly account for the competition between current and accumulated historical spectral demands. More importantly, SPAR consistently and substantially outperforms all three methods across the evaluated benchmarks (Table 7).

SPAR also offers substantial computational advantages over LoDA and E2-LoRA. Unlike these methods, which involve more elaborate mechanisms such as update-direction search and constraints, meta-learning, or self-distillation, SPAR employs a simple probe-based rank scheduler. As shown in Table 10, SPAR achieves the shortest total training time on all four benchmarks, demonstrating that its spectral-capacity control is not only effective but also considerably more efficient.

We acknowledge that the improvements over LoDA and E2-LoRA are modest in balanced settings. However, under increasingly heterogeneous task streams, SPAR's advantage over LoDA grows from 1.54 to 8.26 percentage points (Table 4), demonstrating when demand-aware capacity allocation becomes particularly valuable.

In essence, the novelty of this work lies not merely in achieving better numbers, but in challenging the prevailing direction-centric perspective, identifying spectral capacity as an alternative mechanism, and demonstrating its effectiveness, efficiency, and advantages under heterogeneous task streams.

We hope these clarifications address your concerns and would appreciate it if you could reconsider the paper in light of these points.

Thanks for the authors' response. Although my concerns regarding the novelty of this paper remain, I found and appreciate the results in Table 10. They demonstrate that the proposed method achieves competitive performance with faster training speed, verifying its superior efficiency compared with other baselines. Therefore, I am willing to raise my score.

Thank you for your thoughtful feedback and for acknowledging the efficiency advantages of our approach. We appreciate your time and constructive comments.

Comments on the paper

1 comment
Area_Chair_SqSSOpenReview ↗

[Reminder] Please Join the Discussion

Dear Reviewers UNqu and z71Z,

The authors have submitted their responses. Please read them and update your reviews if appropriate. It would be helpful to share your thoughts early to allow enough time for a productive discussion. Thank you!

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.