acceptodds
Under review as a conference paper at ICLR 2027

Bilateral spectral Low-Rank Adaptation for Class-Incremental Learning

Abstract

Merging sequential low-rank updates into a shared residual enables LoRA-based class-incremental learning (CIL) without retaining task-specific adapters, but introduces severe cross-task interference. Prior methods mitigate forgetting through subspace or gradient constraints, while recent spectral approaches mainly regulate task-local adaptation spectra, leaving the joint input–output spectral structure of the accumulated residual underexplored. In this paper, we conduct a spectral analysis of the accumulated residual to diagnose this interference. We reveal that (i) incoming updates increasingly overlap with historical left and right singular subspaces; and (ii) bilateral-overlap components are substantially more interference-prone than one-sided or complementary components. Based on these observations, we introduce SF-LoRA, a bilateral spectral filtering framework that achieves a better stability-plasticity trade-off. SF-LoRA constructs a compact spectral memory through energy-density-guided truncation, preserving informative modes without unnecessarily sacrificing plastic capacity. At each optimization step, SF-LoRA applies a closed-form proximal correction that selectively attenuates interference-prone bilateral components while granting new tasks full access to complementary orthogonal spaces. Across ImageNet-R, ImageNet-A, CIFAR-100, and CUB-200, SF-LoRA establishes a new state of the art, lifting last accuracy by up to 1.60 percentage points while reducing forgetting without retaining task-specific adapters.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA, Class-Incremental Learning,PEFT

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A6GnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forgetting caused by sequentially merging LoRA updates in replay-free class-incremental learning. The authors divide updates into four regions based on their left and right singular subspaces relative to the historical cumulative residual, and find that normalized forgetting is higher in the region of overlap on both sides. Based on this, they propose SF-LoRA, which constructs spectral memory through energy-density truncation and applies bilateral soft filtering weighted by spectral importance to the LoRA factors after each optimization step. The method achieves excellent experimental results.

Critical strengths
  1. The main thread is clear, the paper is concise, and the method is straightforward; the closed-form derivation is consistent with the low-rank implementation.

  2. The experimental results are good.

Critical weaknesses
  1. The paper’s main novelty is bilateral spectral filtering. However, to my knowledge, OPLoRA already uses bilateral singular subspace projection, while TailLoR uses soft penalties weighted by spectral importance. The authors should clarify how their method differs from these approaches.

  2. The paper does not sufficiently explain its core mechanism: the current analysis does not adequately distinguish the effectiveness of “bilateral structure” from that of “stronger contraction.” NFR first applies updates to each region according to their original magnitudes and then divides by the norm. Since changes in accuracy are not a linear function of the norm, this post hoc normalization is not equivalent to an intervention of equal strength. The unilateral and bilateral ablations also do not report controls matched for overall contraction strength, so it remains unclear whether some of the gains are due to stronger regularization.

3.This method performs only moderately on long tasks such as ImageNet-R with T=20. As far as I know, several papers from 2026 have already exceeded 80% in final accuracy: GR-LoRA, LoDA.

GR-LoRA: GR-LoRA: Gradient-Recycling Low-Rank Adaptation for Class-Incremental Learning.

LoDA: Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning.

Questions for authors
  1. Clarifying novelty and providing additional evidence: After applying the same history-accumulated residual to the existing soft spectral penalty, does SF-LoRA’s specific filtering form still offer an advantage? Can you provide a comparison under the same spectral memory, LoRA rank, and tuning budget?

  2. Providing additional mechanistic evidence: After matching update norms by layer across the four regions and scanning perturbation magnitudes, do the two-sided regions still cause more forgetting? After matching overall contraction strength, does two-sided filtering still outperform one-sided filtering?

Despite the issues above, I still consider this a solid paper. A strong rebuttal could persuade me to raise my score to 3.

5 comments
AuthorsOpenReview ↗

Response to Reviewer A6Gn (1/2)

We sincerely thank the reviewer for the thoughtful and constructive review. We appreciate your recognition of SF-LoRA’s simplicity and strong empirical performance. We have added experiments and explanations to address your concerns.

W1 & Q1: Differences from OPLoRA and TailLoR

Our contribution lies in filtering against the accumulated historical residual with a multiplicative proximal operator. The distinction has three parts.

1. Protecting accumulated adaptation. OPLoRA and TailLoR use a fixed pretrained reference, whereas SF-LoRA builds memory from the evolving residual Ht−1=Wt−1−W0H_{t-1}=W_{t-1}-W_0. The protected directions therefore reflect adaptations learned during the task sequence, not just the initial model.

2. Retaining access to historical directions. Unlike hard projection, SF-LoRA continuously attenuates overlap according to spectral importance on both sides, allowing shared directions to be reused. Compared with TailLoR's maximum-based penalty in the pretrained basis, this ties protection to the evolving adaptation history.

3. Preserving low-rank training. The exact solution X+=(TLB)(ATR)X^+=(T_LB)(AT_R) filters the LoRA factors directly, avoiding a dense update and preserving rank at finite nonnegative strengths. This allows bilateral protection at each training step without leaving the low-rank parameterization.

Following your suggestion, we compared hard projection, soft penalty, and SF-Prox using the same historical memory and LoRA rank within each dataset. The memory comes from a common Vanilla LoRA trajectory, and the two soft variants use the same regularization coefficient. The final accuracies (%) on the two datasets are shown below.

MethodImageNet-AImageNet-R
Hard projection62.4079.55
Soft penalty62.6179.32
SF-Prox63.7980.40

The corresponding average accuracies (%) are:

MethodImageNet-AImageNet-R
Hard projection72.4986.27
Soft penalty72.5386.02
SF-Prox73.2786.63

Entries are final/average accuracy (%).

With the same historical memory, SF-Prox improves final accuracy over Soft penalty by 1.18 pp on ImageNet-A and 1.08 pp on ImageNet-R; average accuracy improves by 0.74 and 0.61 pp, respectively. These gains show that the choice of filtering rule matters even when the historical reference is fixed.

AuthorsOpenReview ↗

Response to Reviewer A6Gn (2/2)

Response to Reviewer A6Gn (2/2)

W2 & Q2: Separating bilateral structure from correction strength

We agree that the original NFR analysis and one-sided ablations do not fully separate these effects. We added two comparisons to address them directly.

First, we matched the perturbation norms across regions. Each regional update was rescaled to the same norm within each layer and applied separately to the same historical model, with the classifier and old-class samples fixed. The results below show old-class accuracy drops on ImageNet-A in percentage points (pp).

StrengthOutput-onlyInput-onlyComplementaryBilateral
s=0.1250.23 ± 0.060.19 ± 0.020.15 ± 0.060.26 ± 0.08
s=0.250.38 ± 0.270.24 ± 0.080.22 ± 0.070.47 ± 0.22
s=0.50.56 ± 0.320.26 ± 0.200.20 ± 0.170.57 ± 0.17
s=10.65 ± 0.290.42 ± 0.150.21 ± 0.060.86 ± 0.21
s=21.19 ± 0.333.07 ± 0.490.91 ± 0.316.21 ± 0.48

Here, ss is the perturbation norm relative to the full update in each layer. Results are mean ± standard deviation across three runs.

The bilateral region has the largest mean accuracy drop at all five strengths. The differences are small at lower strengths and clearest at s=2s=2. Its average L2 representation drift is also largest: 10.63, compared with 7.47/9.34/6.57 for the output-only/input-only/complementary regions.

Second, we matched the amount of correction across filters. We applied Left, Right, Additive, and Multiplicative filtering to the same update with the same historical memory. We measured correction strength as ρ=∥X~−X∥F/∥X∥F\rho=\|\widetilde X-X\|_F/\|X\|_F and adjusted each filter separately to match this quantity within each layer. The changes in old-class NLL on ImageNet-A are shown below.

ρ\rhoLeftRightAdditiveMultiplicative
0.100.00169−0.00103−0.00123−0.00424
0.20−0.00878−0.01228−0.00890−0.01778
0.40−0.00094−0.01402−0.01214−0.01942

Values are changes in old-class NLL relative to applying the unfiltered update, in nats/sample; lower is better. These are interventions on fixed updates.

Multiplicative filtering yields the lowest old-class NLL at all three correction levels. At ρ=0.20\rho=0.20, for example, it reduces NLL by 0.01778 relative to the unfiltered update, compared with 0.00878 for Left, 0.01228 for Right, and 0.00890 for Additive. The improvement on these fixed updates remains when the correction magnitude is matched, supporting the importance of where the correction is applied.

W3: Comparison with results above 80% on ImageNet-R

Thank you for pointing out these results. On 20-task ImageNet-R, both LoDA's 81.13% and GR-LoRA's 80.23% include post-task classifier alignment (CA). Our submitted paper already compares with LoDA without CA (78.96%); SF-LoRA achieves 79.45% under the same setting, higher by 0.49 pp.

CA is an additional enhancement technique that recalibrates the prediction head after each task using class-level feature statistics to reduce old/new-class bias. Results with and without CA should therefore be compared separately. We reran GR-LoRA and E²-LoRA without CA to match SF-LoRA's setting, retaining ordinary classifier-head training. The final accuracies (%) on ImageNet-R are shown below.

Method5 tasks10 tasks20 tasks
GR-LoRA77.57 ± 0.5274.97 ± 0.6168.37 ± 0.57
E²-LoRA81.43 ± 0.4178.73 ± 0.4875.68 ± 0.38
SF-LoRA83.66 ± 0.1982.48 ± 0.1779.45 ± 0.22

The corresponding average accuracies (%) are:

Method5 tasks10 tasks20 tasks
GR-LoRA83.70 ± 0.4481.80 ± 0.3975.72 ± 0.46
E²-LoRA85.73 ± 0.6384.40 ± 0.5582.54 ± 0.60
SF-LoRA87.87 ± 0.3587.64 ± 0.4285.53 ± 0.15

Results are mean ± standard deviation. SF-LoRA achieves the highest mean accuracy in all three settings, with a final-accuracy improvement of 3.77 pp over E²-LoRA at 20 tasks.

We also extended the evaluation to 50 and 100 tasks, using the same no-CA setting. The results below report accuracy (%).

Method50T Final50T Average100T Final100T Average
GR-LoRA62.8271.0350.3457.34
E²-LoRA66.9074.4351.3859.39
SF-LoRA72.7280.0259.0064.92

SF-LoRA retains its advantage as the sequence grows, improving final accuracy over E²-LoRA by 5.82 pp at 50 tasks and 7.62 pp at 100 tasks.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

CA is a computationally inexpensive operation. Could you provide the results of your method with CA added?

Reviewer_ZmojOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SF-LoRA addresses catastrophic forgetting in class-incremental learning through bilateral spectral filtering of sequentially merged LoRA updates. By decomposing incoming updates along the left and right singular subspaces of accumulated residuals, SF-LoRA identifies bilateral overlap as a major source of interference. An energy-density-based truncation scheme constructs compact spectral memory, while a closed-form proximal operator suppresses interference-prone directions. Experiments across four benchmarks report consistent accuracy improvements over existing methods.

Critical strengths
  1. Clear motivation and formulation. Bilateral spectral decomposition provides an intuitive perspective on interference between incoming updates and accumulated historical residuals.

  2. Efficient optimization. Closed-form spectral filtering operates directly on LoRA factors without materializing dense update matrices.

  3. Encouraging empirical results. Consistent improvements across multiple CIL benchmarks, supported by ablations and computational analyses, demonstrate practical potential.

Critical weaknesses
  1. EBLoRA (ICML 2026) investigates spectral imbalance in continual adaptation, E²-LoRA (ICML 2026) exploits spectral energy for knowledge preservation and capacity allocation, and JANUS-LoRA (ICML 2026) employs historical-subspace constraints with closed-form update rectification. Related work on spectral over-accumulation in model merging also examines interference arising from overlapping spectral directions in accumulated updates. Qualitative comparisons in the related-work discussion do not adequately justify the claimed novelty, while direct empirical comparisons with closely related spectral methods are missing.

  2. Fig. 2 attributes stronger forgetting to bilateral spectral overlap using a Normalized Forgetting Rate (NFR). However, NFR divides accuracy degradation under the original regional update by its Frobenius norm, rather than comparing different spectral regions under matched perturbation magnitudes. Given the nonlinear relationship between parameter perturbations and classification accuracy, such normalization does not necessarily establish intrinsic interference severity.

  3. Moreover, large singular values of accumulated weight residuals do not necessarily indicate high sensitivity of historical task performance. Bilateral subspaces also contain cross-mode directions uivj⊤u_i v_j^\top with i≠ji\neq j, whose importance is not justified by the singular-value decomposition alone.

  4. Proposition 3.1 provides a valid closed-form solution to the constructed quadratic objective but offers no guarantee regarding historical loss preservation or forgetting reduction. In particular, the γs2\gamma_s^2 interaction term facilitates factorized bilateral shrinkage without establishing why such multiplicative regularization is preferable for continual learning.

  5. Existing left-only and right-only ablations cannot isolate the benefit of bilateral interaction from that of regularizing both spectral subspaces. An additive two-sided proximal baseline without the interaction term, evaluated under comparable shrinkage strengths, is necessary to establish its independent contribution.

Questions for authors
  1. What specific advantage does joint bilateral filtering provide over existing spectral importance weighting and historical-subspace regularization methods? Can controlled comparisons with EBLoRA, E²-LoRA, or related approaches substantiate the methodological distinction?
  2. Does bilateral overlap remain more interference-prone when all spectral regions are evaluated under equal-norm perturbations? How do historical loss and representation drift vary across these regions?
  3. Does energy-density truncation outperform conventional spectral truncation under identical memory budgets, and how closely do retained singular directions correlate with historical task importance?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Zmoj (1/3)

We sincerely thank the reviewer for the detailed and constructive review. We appreciate your recognition of SF-LoRA’s efficiency and practical potential. We have added experiments and clarifications to address your questions.

W1 & Q1: Contribution of joint bilateral filtering

SF-LoRA combines three choices to protect accumulated adaptation while allowing new tasks to reuse historical directions.

1. An evolving historical reference. We build spectral memory from Ht−1=Wt−1−W0H_{t-1}=W_{t-1}-W_0, so the protection target follows adaptations accumulated across tasks. This differs from EBLoRA's task-update spectral balancing and E²-LoRA's output-drift analysis by considering joint input-output overlap with the accumulated parameter residual.

2. Selective protection with continued reuse. Compared with historical orthogonality constraints such as JANUS-LoRA's, SF-LoRA continuously adjusts shrinkage according to importance on both sides. Bilateral components receive stronger attenuation, while shared directions remain available for new-task adaptation.

3. Exact filtering on LoRA factors. The solution X+=(TLB)(ATR)X^+=(T_LB)(AT_R) avoids constructing a dense update and preserves rank at finite nonnegative strengths. Historical protection can therefore be applied during low-rank training, rather than only calibrating weights after model merging.

We also reran E²-LoRA and GR-LoRA under our image class-incremental protocol. Classifier alignment (CA) recalibrates the classifier after each task using class-level feature statistics to reduce old/new-class prediction bias. As SF-LoRA does not use this stage, we compare no-CA variants while retaining ordinary classifier-head training. This keeps the comparison focused on continual adaptation without additional classifier recalibration. The final accuracies (%) on ImageNet-R are shown below.

Method5 tasks10 tasks20 tasks
JANUS-LoRA77.19 ± 0.3975.78 ± 0.1671.57 ± 0.45
GR-LoRA77.57 ± 0.5274.97 ± 0.6168.37 ± 0.57
E²-LoRA81.43 ± 0.4178.73 ± 0.4875.68 ± 0.38
SF-LoRA83.66 ± 0.1982.48 ± 0.1779.45 ± 0.22

The corresponding average accuracies (%) are:

Method5 tasks10 tasks20 tasks
JANUS-LoRA82.52 ± 0.3581.64 ± 0.2777.11 ± 0.35
GR-LoRA83.70 ± 0.4481.80 ± 0.3975.72 ± 0.46
E²-LoRA85.73 ± 0.6384.40 ± 0.5582.54 ± 0.60
SF-LoRA87.87 ± 0.3587.64 ± 0.4285.53 ± 0.15

SF-LoRA achieves higher final and average accuracy than both rerun no-CA variants in all three settings. At 20 tasks, the gains over E²-LoRA without CA are 3.77/2.99 pp. EBLoRA's LLaVA-based multimodal setting requires additional adaptation to our image protocol, so we prioritized the directly comparable E²-LoRA experiments. To examine the filtering mechanism itself, we also compare additive and multiplicative filtering at matched correction magnitudes in W4 & W5 below.

AuthorsOpenReview ↗

Response to Reviewer Zmoj (2/3)

W2 & Q2: Bilateral sensitivity under equal-norm perturbations

Following your suggestion, we rescaled the four regional updates to the same norm within each layer and applied them separately to the same historical model, keeping the classifier and old-class samples fixed. The results below show old-class accuracy drops on ImageNet-A in percentage points (pp).

StrengthOutput-onlyInput-onlyComplementaryBilateral
s=0.1250.23 ± 0.060.19 ± 0.020.15 ± 0.060.26 ± 0.08
s=0.250.38 ± 0.270.24 ± 0.080.22 ± 0.070.47 ± 0.22
s=0.50.56 ± 0.320.26 ± 0.200.20 ± 0.170.57 ± 0.17
s=10.65 ± 0.290.42 ± 0.150.21 ± 0.060.86 ± 0.21
s=21.19 ± 0.333.07 ± 0.490.91 ± 0.316.21 ± 0.48

Here, ss is the perturbation norm relative to the full update in each layer. Accuracy drops are reported as mean ± standard deviation across three runs. We also measured loss change and representation drift, averaged over the five perturbation strengths:

RegionMean NLL changeL2 driftCosine distance
Output-only−0.0004 ± 0.00147.47 ± 0.430.0186 ± 0.0012
Input-only0.0183 ± 0.00469.34 ± 0.680.0304 ± 0.0010
Complementary−0.0017 ± 0.00456.57 ± 0.410.0156 ± 0.0016
Bilateral0.0424 ± 0.002710.63 ± 0.370.0399 ± 0.0017

NLL (negative log-likelihood) measures the model's prediction loss on old-class samples. We report the signed change, NLL after perturbation minus NLL before perturbation (nats/sample), averaged over the five strengths. Cosine distance is one minus cosine similarity.

The bilateral region has the largest mean accuracy drop at every tested strength. The clearest separation is at s=2s=2: 6.21 pp, versus at most 3.07 pp elsewhere; its signed NLL increase is also largest there (0.2541). These results support regional sensitivity differences after controlling perturbation norm.

W3 & Q3: Effectiveness of spectral weighting and truncation

We tested diagonal spectral energy and the off-diagonal weight αiαj\alpha_i\alpha_j against historical sensitivity, using equal-magnitude positive and negative perturbations along unit-norm directions. Loss sensitivity is measured through central finite differences, alongside representation-drift sensitivity. The Spearman correlations with these sensitivity measures are shown below.

Directions and predictorA NLLA driftR NLLR drift
Diagonal: spectral energy0.720.790.730.82
Off-diagonal: αiαj\alpha_i\alpha_j0.600.670.660.73
All retained: αiαj\alpha_i\alpha_j0.640.720.690.78

A/R denote ImageNet-A/ImageNet-R. For off-diagonal directions, the joint weight αiαj\alpha_i\alpha_j correlates with loss sensitivity at 0.60/0.66 and representation sensitivity at 0.67/0.73. These results support weighting cross-mode overlap according to both historical directions, beyond considering diagonal singular modes alone.

We further compared truncation rules at equal spectral-memory budgets, varying only the allocation of retained directions across layers. The final accuracies (%) on ImageNet-A are shown below.

Budget KFixedCumulativeDensity
861.6262.5063.27
1662.5362.0163.50
3262.7463.1963.96

The corresponding average accuracies (%) are:

Budget KFixedCumulativeDensity
872.5372.2172.68
1672.4672.8273.09
3273.0172.4473.50

KK is the average number of retained directions per layer. All methods use the same total number of directions, representation, and precision. Density achieves higher final and average accuracy at every budget, improving final accuracy over the stronger alternative by 0.77/0.97/0.77 pp. Thus, its benefit persists when storage is matched.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

AuthorsOpenReview ↗

Response to Reviewer Zmoj (3/3)

W4 & W5: Contribution of the interaction term

We agree that Proposition 3.1 alone does not establish a forgetting guarantee. It gives the exact solution to our proximal objective and shows that the interaction term enables factorized filtering, X+=(TLB)(ATR)X^+=(T_LB)(AT_R), without constructing the full update matrix. At finite nonnegative strengths, this operation preserves the update's rank.

To test the interaction term empirically, we compare against an additive objective that retains both single-sided constraints and removes only their interaction. For a bilateral direction, the respective shrinkage coefficients are

cadd=11+γadd(αi+αj),cmul=1(1+γmulαi)(1+γmulαj).c_{\mathrm{add}}=\frac{1}{1+\gamma_{\mathrm{add}}(\alpha_i+\alpha_j)},\qquad c_{\mathrm{mul}}=\frac{1}{(1+\gamma_{\mathrm{mul}}\alpha_i)(1+\gamma_{\mathrm{mul}}\alpha_j)}.

We keep the historical checkpoint, spectral memory, and unfiltered update fixed, and calibrate the two strengths separately to match the actual correction magnitude, ρ=∥X~−X∥F/∥X∥F\rho=\|\widetilde X-X\|_F/\|X\|_F. The changes in old-class NLL on ImageNet-A are shown below.

ρ\rhoAdditive ↓Multiplicative ↓
0.10−0.00123−0.00424
0.20−0.00890−0.01778
0.40−0.01214−0.01942

Each value is old-class NLL after the filtered update minus that after the unfiltered update, in nats/sample; lower is better.

At the same correction magnitudes, multiplicative filtering lowers old-class NLL relative to additive filtering by 0.00301, 0.00888, and 0.00728 nats/sample at ρ=0.10,0.20,\rho=0.10, 0.20, and 0.400.40, respectively. Since both filters use the same historical subspaces and unfiltered updates, these interventions support the multiplicative rule's benefit for old-class loss beyond simply applying a larger correction.

Reviewer_GZFBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SF-LoRA for class-incremental learning with merged low-rank updates. It constructs spectral memory from the accumulated residual and applies importance-weighted bilateral filtering to reduce interference. Experiments show consistent improvements across four benchmarks.

Critical strengths

1.The distinction between bilateral, one-sided, and complementary overlap provides a useful perspective on interference between sequential updates. 2. The filter has a clear closed-form derivation and can be implemented directly on LoRA factors without constructing dense updates. 3. Results consistently improve over the reported baselines. One-sided filtering ablations and forgetting measurements support the complete design.

Critical weaknesses
  1. The interference diagnosis needs stronger controls. Normalized forgetting is measured by dividing accuracy loss by update magnitude after evaluation. Since accuracy responds nonlinearly to perturbations, this does not fully control for magnitude differences. Evaluating regional updates at matched norms and multiple scales would strengthen the central claim.

  2. Spectral energy is not necessarily task importance. Large singular values of the accumulated residual may reflect update magnitude rather than importance for historical predictions. Conversely, cancellation between task updates may hide important directions. The reliability of this historical surrogate needs further investigation.

  3. Comparisons with closely related spectral methods are missing. EBLoRA, E²-LoRA, and SCLoRA are discussed but not empirically evaluated. Differences in their spectral targets do not remove the value of comparisons under a shared protocol.

  4. The benefit of energy-density truncation is insufficiently isolated. The threshold analysis does not directly establish superiority over fixed-rank or cumulative-energy truncation. Comparisons with matched memory budgets would better justify this component.

  5. Practical efficiency and generality need further validation. Table 4 reports computation and spectral-memory storage, but wall-clock time, peak memory during SVD, and access to the original pretrained checkpoint should also be accounted for. All experiments use the same backbone, and several gains over the strongest baseline are modest.

Questions for authors
  1. Does bilateral overlap remain more harmful when regional updates have matched norms?
  2. How does energy-density truncation compare with alternatives at equal storage budgets?
  3. Do the improvements persist across other pretrained backbones and longer task sequences?
3 comments
AuthorsOpenReview ↗

Response to Reviewer GZFB (1/2)

e sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of SF-LoRA’s clear formulation and consistent performance gains. We have added experiments and explanations to address your concerns.

W1 & Q1: Controlling perturbation magnitude

We agree that normalizing accuracy drop by the update norm does not, by itself, rule out a magnitude confound. We therefore rescaled the four regional updates to the same Frobenius norm in each layer, then applied them separately to the same historical checkpoint, keeping the classifier and old-class evaluation samples fixed. The target norm is ss times that of the full current update in the corresponding layer. The old-class accuracy drops on ImageNet-A are shown below.

StrengthOutput-onlyInput-onlyComplementaryBilateral
s=0.1250.23 ± 0.060.19 ± 0.020.15 ± 0.060.26 ± 0.08
s=0.250.38 ± 0.270.24 ± 0.080.22 ± 0.070.47 ± 0.22
s=0.50.56 ± 0.320.26 ± 0.200.20 ± 0.170.57 ± 0.17
s=10.65 ± 0.290.42 ± 0.150.21 ± 0.060.86 ± 0.21
s=21.19 ± 0.333.07 ± 0.490.91 ± 0.316.21 ± 0.48

Accuracy drop is measured in percentage points (pp); entries are mean ± standard deviation across three runs.

The mean L2 representation drift over the five perturbation strengths is shown below;

RegionMean L2 representation drift
Output-only7.47 ± 0.43
Input-only9.34 ± 0.68
Complementary6.57 ± 0.41
Bilateral10.63 ± 0.37

The bilateral region has the largest mean drop at every tested strength. At s=2s=2, its drop is 6.21 pp, compared with at most 3.07 pp for the other regions. It also has the largest mean L2 representation drift. On ImageNet-A, the larger mean accuracy drop therefore remains after matching the regional update norms.

W2: Spectral energy as an importance proxy

We tested whether spectral energy predicts sensitivity on historical data. We applied equal-magnitude positive and negative perturbations along unit-norm singular directions and measured changes in old-class negative log-likelihood (NLL) and representations. The Spearman correlations between spectral energy and these sensitivity measures are shown below.

DatasetLoss sensitivityRepresentation sensitivity
ImageNet-A0.720.79
ImageNet-R0.730.82

Spectral energy correlates positively with both old-class loss sensitivity (0.72/0.73 on ImageNet-A/R) and representation sensitivity (0.79/0.82). This supports assigning stronger protection to higher-energy historical directions.

W3: Comparisons with related spectral methods

We have added comparisons with E²-LoRA and GR-LoRA under our image class-incremental protocol. Classifier alignment (CA) recalibrates the classifier after each task using class-level feature statistics to reduce prediction bias between old and new classes. Since SF-LoRA does not use this step, we evaluated variants without CA under the same no-recalibration setting, while retaining ordinary classifier-head training. The final accuracies (%) on ImageNet-R are shown below.

Method5 tasks10 tasks20 tasks
JANUS-LoRA77.19 ± 0.3975.78 ± 0.1671.57 ± 0.45
GR-LoRA77.57 ± 0.5274.97 ± 0.6168.37 ± 0.57
E²-LoRA81.43 ± 0.4178.73 ± 0.4875.68 ± 0.38
SF-LoRA83.66 ± 0.1982.48 ± 0.1779.45 ± 0.22

The corresponding average accuracies (%) are:

Method5 tasks10 tasks20 tasks
JANUS-LoRA82.52 ± 0.3581.64 ± 0.2777.11 ± 0.35
GR-LoRA83.70 ± 0.4481.80 ± 0.3975.72 ± 0.46
E²-LoRA85.73 ± 0.6384.40 ± 0.5582.54 ± 0.60
SF-LoRA87.87 ± 0.3587.64 ± 0.4285.53 ± 0.15

SF-LoRA achieves higher final and average accuracy than both rerun variants in all three settings. Its final-accuracy gains over E²-LoRA without CA are 2.23/3.75/3.77 pp for 5/10/20 tasks.

EBLoRA and SCLoRA evaluate LLaVA multimodal continual adaptation and language models, respectively. Transferring them to our ViT image-classification setting requires additional adaptation, so we prioritized a direct comparison with E²-LoRA and will clarify the methodological distinctions from the other spectral approaches.

AuthorsOpenReview ↗

Response to Reviewer GZFB (2/2)

W4 & Q2: Comparing truncation at the same storage budget

To separate the effect of direction allocation from storage capacity, we compared fixed-rank, cumulative-energy, and energy-density allocation at the same total spectral-memory budget, using the same representation and numerical precision. The final accuracies (%) on ImageNet-A are shown below.

Budget KFixedCumulativeDensity
861.6262.5063.27
1662.5362.0163.50
3262.7463.1963.96

The corresponding average accuracies (%) are:

Budget KFixedCumulativeDensity
872.5372.2172.68
1672.4672.8273.09
3273.0172.4473.50

KK denotes the average number of retained directions per layer. The total number of directions is identical across methods; only their allocation across layers differs.

Density allocation gives the highest final and average accuracy at all three budgets, improving final accuracy over the stronger alternative by 0.77/0.97/0.77 pp. The benefit therefore persists when spectral-memory capacity is held fixed.

W5 & Q3: Practical cost and broader evaluation

Runtime and memory. We measured full-sequence wall-clock time on 10-task ImageNet-R, together with the increase in peak allocated GPU memory during SVD relative to immediately before the call.

MethodTotal timeSVD peak increment (MiB)
InfLoRA1h46m31.56
LoDA1h43m39.24
SF-LoRA1h38m36.34

On this 10-task ImageNet-R run, SF-LoRA takes 8 minutes less than InfLoRA and 5 minutes less than LoDA. Its SVD memory increment is 36.34 MiB, between the two baselines.

We also acknowledge the cost of accessing W0W_0 to construct Ht−1=Wt−1−W0H_{t-1}=W_{t-1}-W_0. Our implementation uses one locally cached checkpoint of 391.43 MiB, shared across the task sequence and not reread at every optimization step.

Different pretraining schemes. We added 10-task ImageNet-R experiments using DINO-1k and iBOT-1k backbones. The final accuracies (%) are shown below.

MethodDINO-1kiBOT-1k
InfLoRA68.3171.84
VPT-NSP268.9673.25
SplitLoRA70.4974.58
SF-LoRA74.2177.33

The corresponding average accuracies (%) are:

MethodDINO-1kiBOT-1k
InfLoRA76.3578.20
VPT-NSP276.2279.65
SplitLoRA78.1581.45
SF-LoRA81.3282.87

SF-LoRA improves over SplitLoRA by 3.72/3.17 pp on DINO-1k and 2.75/1.42 pp on iBOT-1k in final/average accuracy. Thus, the gains on 10-task ImageNet-R also hold for these two self-supervised pretraining schemes.

Longer task sequences. We also evaluated 50- and 100-task ImageNet-R. The results below report accuracy (%).

Method50T Last ↑50T Avg ↑100T Last ↑100T Avg ↑
GR-LoRA62.8271.0350.3457.34
E²-LoRA66.9074.4351.3859.39
SF-LoRA72.7280.0259.0064.92

The final/average accuracy gains over E²-LoRA without CA are 5.82/5.59 pp for 50 tasks and 7.62/5.53 pp for 100 tasks, showing that the advantage in this setting persists over longer sequences.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.