- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies forgetting caused by sequentially merging LoRA updates in replay-free class-incremental learning. The authors divide updates into four regions based on their left and right singular subspaces relative to the historical cumulative residual, and find that normalized forgetting is higher in the region of overlap on both sides. Based on this, they propose SF-LoRA, which constructs spectral memory through energy-density truncation and applies bilateral soft filtering weighted by spectral importance to the LoRA factors after each optimization step. The method achieves excellent experimental results.
Critical strengths
-
The main thread is clear, the paper is concise, and the method is straightforward; the closed-form derivation is consistent with the low-rank implementation.
-
The experimental results are good.
Critical weaknesses
-
The paper’s main novelty is bilateral spectral filtering. However, to my knowledge, OPLoRA already uses bilateral singular subspace projection, while TailLoR uses soft penalties weighted by spectral importance. The authors should clarify how their method differs from these approaches.
-
The paper does not sufficiently explain its core mechanism: the current analysis does not adequately distinguish the effectiveness of “bilateral structure” from that of “stronger contraction.” NFR first applies updates to each region according to their original magnitudes and then divides by the norm. Since changes in accuracy are not a linear function of the norm, this post hoc normalization is not equivalent to an intervention of equal strength. The unilateral and bilateral ablations also do not report controls matched for overall contraction strength, so it remains unclear whether some of the gains are due to stronger regularization.
3.This method performs only moderately on long tasks such as ImageNet-R with T=20. As far as I know, several papers from 2026 have already exceeded 80% in final accuracy: GR-LoRA, LoDA.
GR-LoRA: GR-LoRA: Gradient-Recycling Low-Rank Adaptation for Class-Incremental Learning.
LoDA: Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning.
Questions for authors
-
Clarifying novelty and providing additional evidence: After applying the same history-accumulated residual to the existing soft spectral penalty, does SF-LoRA’s specific filtering form still offer an advantage? Can you provide a comparison under the same spectral memory, LoRA rank, and tuning budget?
-
Providing additional mechanistic evidence: After matching update norms by layer across the four regions and scanning perturbation magnitudes, do the two-sided regions still cause more forgetting? After matching overall contraction strength, does two-sided filtering still outperform one-sided filtering?
Despite the issues above, I still consider this a solid paper. A strong rebuttal could persuade me to raise my score to 3.
5 comments
Response to Reviewer A6Gn (2/2)
Response to Reviewer A6Gn (2/2)
W2 & Q2: Separating bilateral structure from correction strength
We agree that the original NFR analysis and one-sided ablations do not fully separate these effects. We added two comparisons to address them directly.
First, we matched the perturbation norms across regions. Each regional update was rescaled to the same norm within each layer and applied separately to the same historical model, with the classifier and old-class samples fixed. The results below show old-class accuracy drops on ImageNet-A in percentage points (pp).
| Strength | Output-only | Input-only | Complementary | Bilateral |
|---|---|---|---|---|
| s=0.125 | 0.23 ± 0.06 | 0.19 ± 0.02 | 0.15 ± 0.06 | 0.26 ± 0.08 |
| s=0.25 | 0.38 ± 0.27 | 0.24 ± 0.08 | 0.22 ± 0.07 | 0.47 ± 0.22 |
| s=0.5 | 0.56 ± 0.32 | 0.26 ± 0.20 | 0.20 ± 0.17 | 0.57 ± 0.17 |
| s=1 | 0.65 ± 0.29 | 0.42 ± 0.15 | 0.21 ± 0.06 | 0.86 ± 0.21 |
| s=2 | 1.19 ± 0.33 | 3.07 ± 0.49 | 0.91 ± 0.31 | 6.21 ± 0.48 |
Here, is the perturbation norm relative to the full update in each layer. Results are mean ± standard deviation across three runs.
The bilateral region has the largest mean accuracy drop at all five strengths. The differences are small at lower strengths and clearest at . Its average L2 representation drift is also largest: 10.63, compared with 7.47/9.34/6.57 for the output-only/input-only/complementary regions.
Second, we matched the amount of correction across filters. We applied Left, Right, Additive, and Multiplicative filtering to the same update with the same historical memory. We measured correction strength as and adjusted each filter separately to match this quantity within each layer. The changes in old-class NLL on ImageNet-A are shown below.
| Left | Right | Additive | Multiplicative | |
|---|---|---|---|---|
| 0.10 | 0.00169 | −0.00103 | −0.00123 | −0.00424 |
| 0.20 | −0.00878 | −0.01228 | −0.00890 | −0.01778 |
| 0.40 | −0.00094 | −0.01402 | −0.01214 | −0.01942 |
Values are changes in old-class NLL relative to applying the unfiltered update, in nats/sample; lower is better. These are interventions on fixed updates.
Multiplicative filtering yields the lowest old-class NLL at all three correction levels. At , for example, it reduces NLL by 0.01778 relative to the unfiltered update, compared with 0.00878 for Left, 0.01228 for Right, and 0.00890 for Additive. The improvement on these fixed updates remains when the correction magnitude is matched, supporting the importance of where the correction is applied.
W3: Comparison with results above 80% on ImageNet-R
Thank you for pointing out these results. On 20-task ImageNet-R, both LoDA's 81.13% and GR-LoRA's 80.23% include post-task classifier alignment (CA). Our submitted paper already compares with LoDA without CA (78.96%); SF-LoRA achieves 79.45% under the same setting, higher by 0.49 pp.
CA is an additional enhancement technique that recalibrates the prediction head after each task using class-level feature statistics to reduce old/new-class bias. Results with and without CA should therefore be compared separately. We reran GR-LoRA and E²-LoRA without CA to match SF-LoRA's setting, retaining ordinary classifier-head training. The final accuracies (%) on ImageNet-R are shown below.
| Method | 5 tasks | 10 tasks | 20 tasks |
|---|---|---|---|
| GR-LoRA | 77.57 ± 0.52 | 74.97 ± 0.61 | 68.37 ± 0.57 |
| E²-LoRA | 81.43 ± 0.41 | 78.73 ± 0.48 | 75.68 ± 0.38 |
| SF-LoRA | 83.66 ± 0.19 | 82.48 ± 0.17 | 79.45 ± 0.22 |
The corresponding average accuracies (%) are:
| Method | 5 tasks | 10 tasks | 20 tasks |
|---|---|---|---|
| GR-LoRA | 83.70 ± 0.44 | 81.80 ± 0.39 | 75.72 ± 0.46 |
| E²-LoRA | 85.73 ± 0.63 | 84.40 ± 0.55 | 82.54 ± 0.60 |
| SF-LoRA | 87.87 ± 0.35 | 87.64 ± 0.42 | 85.53 ± 0.15 |
Results are mean ± standard deviation. SF-LoRA achieves the highest mean accuracy in all three settings, with a final-accuracy improvement of 3.77 pp over E²-LoRA at 20 tasks.
We also extended the evaluation to 50 and 100 tasks, using the same no-CA setting. The results below report accuracy (%).
| Method | 50T Final | 50T Average | 100T Final | 100T Average |
|---|---|---|---|---|
| GR-LoRA | 62.82 | 71.03 | 50.34 | 57.34 |
| E²-LoRA | 66.90 | 74.43 | 51.38 | 59.39 |
| SF-LoRA | 72.72 | 80.02 | 59.00 | 64.92 |
SF-LoRA retains its advantage as the sequence grows, improving final accuracy over E²-LoRA by 5.82 pp at 50 tasks and 7.62 pp at 100 tasks.
We sincerely appreciate your valuable feedback, which has helped us improve the clarity and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.
CA is a computationally inexpensive operation. Could you provide the results of your method with CA added?
Response to Reviewer A6Gn (1/2)
We sincerely thank the reviewer for the thoughtful and constructive review. We appreciate your recognition of SF-LoRA’s simplicity and strong empirical performance. We have added experiments and explanations to address your concerns.
W1 & Q1: Differences from OPLoRA and TailLoR
Our contribution lies in filtering against the accumulated historical residual with a multiplicative proximal operator. The distinction has three parts.
1. Protecting accumulated adaptation. OPLoRA and TailLoR use a fixed pretrained reference, whereas SF-LoRA builds memory from the evolving residual Ht−1=Wt−1−W0. The protected directions therefore reflect adaptations learned during the task sequence, not just the initial model.
2. Retaining access to historical directions. Unlike hard projection, SF-LoRA continuously attenuates overlap according to spectral importance on both sides, allowing shared directions to be reused. Compared with TailLoR's maximum-based penalty in the pretrained basis, this ties protection to the evolving adaptation history.
3. Preserving low-rank training. The exact solution X+=(TLB)(ATR) filters the LoRA factors directly, avoiding a dense update and preserving rank at finite nonnegative strengths. This allows bilateral protection at each training step without leaving the low-rank parameterization.
Following your suggestion, we compared hard projection, soft penalty, and SF-Prox using the same historical memory and LoRA rank within each dataset. The memory comes from a common Vanilla LoRA trajectory, and the two soft variants use the same regularization coefficient. The final accuracies (%) on the two datasets are shown below.
The corresponding average accuracies (%) are:
Entries are final/average accuracy (%).
With the same historical memory, SF-Prox improves final accuracy over Soft penalty by 1.18 pp on ImageNet-A and 1.08 pp on ImageNet-R; average accuracy improves by 0.74 and 0.61 pp, respectively. These gains show that the choice of filtering rule matters even when the historical reference is fixed.