- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies continual learning with pre-trained models and proposes Incremental Spectral Merging (ISM). After each task, it keeps the main singular directions of the LoRA update and takes a small step toward the new module. It then adjusts old-class feature statistics and realigns the classifier. The method performs well across several benchmarks while keeping a single merged adapter for inference.
Critical strengths
-
The approach is fairly simple and practical. It incorporates new tasks without keeping a growing collection of adapters or requiring task identity at test time.
-
The experiments cover several datasets and include robustness tests. The comparison that changes only the merging rule is particularly useful, since it helps separate the merge’s contribution from the rest of the pipeline.
Critical weaknesses
-
In Figure 2, when , the accuracy varies by less than 0.25 percentage points across truncation levels, and using no truncation gives a very similar result. At least in this experiment, taking a smaller update seems to matter more. Since spectral truncation is the central contribution, the paper needs stronger evidence of what it adds over simple damped interpolation.
-
The bound mainly shows that limiting model changes helps control the increase in loss. However, its spectral-norm term depends on the largest singular value, so removing smaller singular components does not reduce that term. This supports using a small update coefficient, but does not directly explain the extra benefit of truncation or why low-energy directions can safely be discarded.
-
The number of LoRA modules stays fixed, but the method also stores a mean and a full covariance matrix for every class. With 768-dimensional features, that means a matrix per class. This storage grows as more classes arrive. The paper should distinguish the fixed adapter storage from the total storage required by the method.
Questions for authors
-
Table 5 uses dataset-specific update coefficients for ISM, while Table 6 says that the other merging rules are tuned on ImageNet-R and their settings are reused elsewhere. Did all methods receive the same tuning budget? How much of ISM’s advantage remains if the other rules are also tuned separately on each dataset? Were these settings selected using an independent validation set?
-
The drift correction is fitted using only the current task’s classes and then applied to all old classes. How do you check that this transformation also works for the old classes? Could you use held-out old-class samples, only for diagnostic evaluation, to compare the errors in estimated means and covariances before and after correction? This would help show whether the correction works and whether its errors accumulate over longer task sequences.
3 comments
Response to Reviewer xkNt (2/2)
Questions
[xkNt-Q1]
Table 5 uses dataset-specific damping for ISM while the other rules are tuned on ImageNet-R only. Same tuning budget? How much advantage remains if the others are tuned per dataset? Independent validation set?
Response. At submission, ISM's and were set per dataset, DARE, DELLA and KnOTS were tuned on ImageNet-R over a grid and reused elsewhere, and task arithmetic, TIES and max-abs ran at unit coefficient; every choice was made on the test split. The revised Table 6 removes the asymmetry: every method is tuned per dataset over four settings (Appendix C, Table 7). Average final / average incremental accuracy over the four benchmarks, submitted against revised Table 6:
| Rule | Submitted Table 6 | Revised Table 6 |
|---|---|---|
| ISM (ours) | 87.06 / 90.40 | 87.00 / 90.44 |
| NUFILT | - | 86.68 / 90.19 |
| Task Arithmetic | 85.45 / 89.90 | 86.54 / 90.25 |
| DELLA | 85.26 / 89.86 | 86.44 / 90.35 |
| TIES | 84.63 / 89.42 | 86.33 / 90.20 |
| DARE | 84.70 / 89.70 | 86.22 / 90.25 |
| Max-abs | 83.11 / 88.77 | 86.18 / 90.08 |
| KnOTS-TIES | 84.90 / 89.55 | 86.10 / 90.10 |
| Model Stock | 85.75 / 89.99 | 85.61 / 90.03 |
ISM stays first on the average and on CIFAR-100, ImageNet-R and Stanford Cars. The settings tried (Table 7 of the revised PDF):
| Merge rule | Tuned parameters | Four settings |
|---|---|---|
| ISM | damping α, energy fraction p | α ∈ {0.1, 0.3} × p ∈ {0.7, 0.9} |
| Model Stock | none | one setting |
| Task Arithmetic | coefficient c | c ∈ {0.3, 0.5, 0.7, 1.0} |
| DELLA | coefficient c, drop rate | c ∈ {0.3, 1.0} × {0.5, 0.9} |
| KnOTS-TIES | coefficient c, kept rank | c ∈ {0.3, 1.0} × {50, 100} % |
| DARE | coefficient c, drop rate | c ∈ {0.3, 1.0} × {0.5, 0.9} |
| TIES | coefficient c, trim | c ∈ {0.3, 1.0} × {20, 100} % |
| Max-abs | coefficient c | c ∈ {0.3, 0.5, 0.7, 1.0} |
| NUFILT | coefficient c, null-space rank | c ∈ {0.1, 0.3} × {32, 64} |
[xkNt-Q2]
The drift correction is fitted on the current task's classes and applied to all old classes. How do you check that it works for old classes? Held-out old-class samples could measure the error of the corrected means and covariances; do errors accumulate over long sequences?
Response. We ran the variant the reviewer implies (Appendix M): the drift map is fitted on a replay buffer of stored training images per class for every class seen so far, the current task included, instead of the current task alone. It leaves the exemplar-free setting and serves as a reference. On ImageNet-R:
| Drift map fitted on | FA / AA | Forgetting |
|---|---|---|
| ISM (ours) | 83.48 / 87.59 | 3.81 |
| Buffer, 5 images per class | 83.49 / 87.61 | 4.61 |
| Buffer, 10 images per class | 83.56 / 87.61 | 4.44 |
| Buffer, 20 images per class | 83.50 / 87.62 | 4.35 |
The buffer changes final accuracy by at most 0.1 points and raises forgetting by 0.5 to 0.8: the map fitted on the current task alone transports the old statistics as well as one fitted on old-class samples.
Thank you for your responses. I would like to raise my score to 3.
Response to Reviewer xkNt (1/2)
We thank the reviewer for the careful and constructive review; we address each point below.
Weaknesses
[xkNt-W1]
Response. This is a fair reading, and Table 4 now holds the evidence: p=1 and p=0.1 at every damping value on ImageNet-R, with forgetting and the accuracy of the oldest and newest tasks; the General response summarises the two regimes. At the step we deploy (α≤0.3) the truncation is nearly inactive, p=0.9 and p=1 are within 0.1. At α≥0.5 the gap grows with the step, up to 1.0 point at α=1, entirely through the old tasks. The lemma of Section 3.5 says why the two knobs act differently. Theorem 1 bounds the worst-case change of the representation by the spectral norm of the step, ασ1, which only damping lowers; the lemma bounds the typical change from below by its Frobenius norm, α∑iσi2, which truncation lowers by removing the tail of singular values while leaving σ1 unchanged. Damping therefore sets how far the deployed module can move at worst, and truncation how much it moves on average, which is what the old-task accuracy responds to. Appendix J tests the remaining question, whether the removed tail acts on the old tasks: at p=0.1 the discarded directions hold 83 % of the update's energy, produce 76 % of its effect on old-class inputs and lie largely inside the old-task feature subspace (overlap 0.71, against 0.86 for the kept directions), which is why removing them protects the old tasks at large steps.
[xkNt-W2]
Response. Correct. The new lemma (Section 3.5, Appendix A.1; see the General response) supplies the missing half: with only layer k changed, E∥ft+1(x)−ft(x)∥≥μkκk∑iσk,i2, so the typical representation change is governed by all singular values and truncation lowers it, while Theorem 1 bounds the worst case through σ1, which damping controls.
[xkNt-W3]
Response. Agreed; the new Discussion section separates them (see the General response). Fixed: the adapter and the merge inputs (8.26M parameters plus one module in transit). Growing: the class statistics, 7682+768≈0.59M numbers per class, i.e. 59M on CIFAR-100, 118M on CUB-200 and ImageNet-R, 116M on Stanford Cars. Appendix K measures two cheaper statistics on ImageNet-R:
The diagonal covariance costs 1.2 points of final accuracy and the mean alone 1.4.