acceptodds
Under review as a conference paper at ICLR 2027

Incremental Spectral Merging with Pre-trained Models

Abstract

Despite much recent progress, continual learning remains unsolved. Every update to a model's weights risks erasing what the model already knows. Model merging is a promising way to address this challenge. Treating each task update as a separate model, merging combines multiple models into one without retraining and without storing data or extra models. Even simple operators such as averaging or maximum-magnitude selection work well in practice. However, there is little theory explaining why the merged model performs well in continual learning. In this paper, we offer a new perspective on model merging for continual learning. Instead of merging all parameters of the low-rank modules used for fine-tuning, we merge only the dominant singular directions of each update, scaled by a damping factor. Our theoretical analysis bounds the loss of the merged model by two terms: the change of the classification head and the weight step between consecutive modules. Spectral merging sets the second term through its damping factor, and realigning the classifier after the merge keeps the first small. Our experiments show that keeping the dominant low-rank information is key to mitigating forgetting. Across four class-incremental benchmarks, our method ranks first in average final accuracy and in robustness to input corruptions, ahead of both prior continual learning methods and published merging rules.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Model Merging, Spectral Normalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xkNtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies continual learning with pre-trained models and proposes Incremental Spectral Merging (ISM). After each task, it keeps the main singular directions of the LoRA update and takes a small step toward the new module. It then adjusts old-class feature statistics and realigns the classifier. The method performs well across several benchmarks while keeping a single merged adapter for inference.

Critical strengths
  1. The approach is fairly simple and practical. It incorporates new tasks without keeping a growing collection of adapters or requiring task identity at test time.

  2. The experiments cover several datasets and include robustness tests. The comparison that changes only the merging rule is particularly useful, since it helps separate the merge’s contribution from the rest of the pipeline.

Critical weaknesses
  1. In Figure 2, when α=0.3\alpha=0.3, the accuracy varies by less than 0.25 percentage points across truncation levels, and using no truncation gives a very similar result. At least in this experiment, taking a smaller update seems to matter more. Since spectral truncation is the central contribution, the paper needs stronger evidence of what it adds over simple damped interpolation.

  2. The bound mainly shows that limiting model changes helps control the increase in loss. However, its spectral-norm term depends on the largest singular value, so removing smaller singular components does not reduce that term. This supports using a small update coefficient, but does not directly explain the extra benefit of truncation or why low-energy directions can safely be discarded.

  3. The number of LoRA modules stays fixed, but the method also stores a mean and a full covariance matrix for every class. With 768-dimensional features, that means a �0��0� matrix per class. This storage grows as more classes arrive. The paper should distinguish the fixed adapter storage from the total storage required by the method.

Questions for authors
  1. Table 5 uses dataset-specific update coefficients for ISM, while Table 6 says that the other merging rules are tuned on ImageNet-R and their settings are reused elsewhere. Did all methods receive the same tuning budget? How much of ISM’s advantage remains if the other rules are also tuned separately on each dataset? Were these settings selected using an independent validation set?

  2. The drift correction is fitted using only the current task’s classes and then applied to all old classes. How do you check that this transformation also works for the old classes? Could you use held-out old-class samples, only for diagnostic evaluation, to compare the errors in estimated means and covariances before and after correction? This would help show whether the correction works and whether its errors accumulate over longer task sequences.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xkNt (1/2)

We thank the reviewer for the careful and constructive review; we address each point below.

Weaknesses

[xkNt-W1]

At α=0.3\alpha = 0.3 accuracy varies by less than 0.25 across truncation levels and no truncation gives a similar result; taking a smaller update seems to matter more. Stronger evidence is needed for what truncation adds over damped interpolation.

Response. This is a fair reading, and Table 4 now holds the evidence: p=1p = 1 and p=0.1p = 0.1 at every damping value on ImageNet-R, with forgetting and the accuracy of the oldest and newest tasks; the General response summarises the two regimes. At the step we deploy (α≤0.3\alpha \le 0.3) the truncation is nearly inactive, p=0.9p = 0.9 and p=1p = 1 are within 0.1. At α≥0.5\alpha \ge 0.5 the gap grows with the step, up to 1.0 point at α=1\alpha = 1, entirely through the old tasks. The lemma of Section 3.5 says why the two knobs act differently. Theorem 1 bounds the worst-case change of the representation by the spectral norm of the step, ασ1\alpha\sigma_1, which only damping lowers; the lemma bounds the typical change from below by its Frobenius norm, α∑iσi2\alpha\sqrt{\sum_i \sigma_i^2}, which truncation lowers by removing the tail of singular values while leaving σ1\sigma_1 unchanged. Damping therefore sets how far the deployed module can move at worst, and truncation how much it moves on average, which is what the old-task accuracy responds to. Appendix J tests the remaining question, whether the removed tail acts on the old tasks: at p=0.1p = 0.1 the discarded directions hold 83 % of the update's energy, produce 76 % of its effect on old-class inputs and lie largely inside the old-task feature subspace (overlap 0.71, against 0.86 for the kept directions), which is why removing them protects the old tasks at large steps.

[xkNt-W2]

The bound's spectral-norm term depends on the largest singular value, so removing small components does not reduce it; it does not explain why low-energy directions can be discarded.

Response. Correct. The new lemma (Section 3.5, Appendix A.1; see the General response) supplies the missing half: with only layer kk changed, E∥ft+1(x)−ft(x)∥≥μkκk∑iσk,i2\mathbb{E}\|f_{t+1}(x) - f_t(x)\| \ge \mu_k \kappa_k \sqrt{\sum_i \sigma_{k,i}^2}, so the typical representation change is governed by all singular values and truncation lowers it, while Theorem 1 bounds the worst case through σ1\sigma_1, which damping controls.

[xkNt-W3]

The method stores a mean and a full covariance per class; the paper should separate the fixed adapter storage from the total storage.

Response. Agreed; the new Discussion section separates them (see the General response). Fixed: the adapter and the merge inputs (8.26M parameters plus one module in transit). Growing: the class statistics, 7682+768≈0.59768^2 + 768 \approx 0.59M numbers per class, i.e. 59M on CIFAR-100, 118M on CUB-200 and ImageNet-R, 116M on Stanford Cars. Appendix K measures two cheaper statistics on ImageNet-R:

Class statisticsNumbers per classFA / AA
ISM (ours)590,59283.48 / 87.59
Mean and diagonal covariance1,53682.31 / 86.54
Mean alone76882.13 / 86.31

The diagonal covariance costs 1.2 points of final accuracy and the mean alone 1.4.

AuthorsOpenReview ↗

Response to Reviewer xkNt (2/2)

Questions

[xkNt-Q1]

Table 5 uses dataset-specific damping for ISM while the other rules are tuned on ImageNet-R only. Same tuning budget? How much advantage remains if the others are tuned per dataset? Independent validation set?

Response. At submission, ISM's α\alpha and λ\lambda were set per dataset, DARE, DELLA and KnOTS were tuned on ImageNet-R over a 2×22 \times 2 grid and reused elsewhere, and task arithmetic, TIES and max-abs ran at unit coefficient; every choice was made on the test split. The revised Table 6 removes the asymmetry: every method is tuned per dataset over four settings (Appendix C, Table 7). Average final / average incremental accuracy over the four benchmarks, submitted against revised Table 6:

RuleSubmitted Table 6Revised Table 6
ISM (ours)87.06 / 90.4087.00 / 90.44
NUFILT-86.68 / 90.19
Task Arithmetic85.45 / 89.9086.54 / 90.25
DELLA85.26 / 89.8686.44 / 90.35
TIES84.63 / 89.4286.33 / 90.20
DARE84.70 / 89.7086.22 / 90.25
Max-abs83.11 / 88.7786.18 / 90.08
KnOTS-TIES84.90 / 89.5586.10 / 90.10
Model Stock85.75 / 89.9985.61 / 90.03

ISM stays first on the average and on CIFAR-100, ImageNet-R and Stanford Cars. The settings tried (Table 7 of the revised PDF):

Merge ruleTuned parametersFour settings
ISMdamping α, energy fraction pα ∈ {0.1, 0.3} × p ∈ {0.7, 0.9}
Model Stocknoneone setting
Task Arithmeticcoefficient cc ∈ {0.3, 0.5, 0.7, 1.0}
DELLAcoefficient c, drop ratec ∈ {0.3, 1.0} × {0.5, 0.9}
KnOTS-TIEScoefficient c, kept rankc ∈ {0.3, 1.0} × {50, 100} %
DAREcoefficient c, drop ratec ∈ {0.3, 1.0} × {0.5, 0.9}
TIEScoefficient c, trimc ∈ {0.3, 1.0} × {20, 100} %
Max-abscoefficient cc ∈ {0.3, 0.5, 0.7, 1.0}
NUFILTcoefficient c, null-space rankc ∈ {0.1, 0.3} × {32, 64}

[xkNt-Q2]

The drift correction is fitted on the current task's classes and applied to all old classes. How do you check that it works for old classes? Held-out old-class samples could measure the error of the corrected means and covariances; do errors accumulate over long sequences?

Response. We ran the variant the reviewer implies (Appendix M): the drift map is fitted on a replay buffer of NN stored training images per class for every class seen so far, the current task included, instead of the current task alone. It leaves the exemplar-free setting and serves as a reference. On ImageNet-R:

Drift map fitted onFA / AAForgetting
ISM (ours)83.48 / 87.593.81
Buffer, 5 images per class83.49 / 87.614.61
Buffer, 10 images per class83.56 / 87.614.44
Buffer, 20 images per class83.50 / 87.624.35

The buffer changes final accuracy by at most 0.1 points and raises forgetting by 0.5 to 0.8: the map fitted on the current task alone transports the old statistics as well as one fitted on old-class samples.

Thank you for your responses. I would like to raise my score to 3.

Reviewer_q6ZZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Incremental Spectral Merging (ISM), a model merging rule designed for sequential continual learning with pre-trained models. Instead of merging all parameters of sequentially fine-tuned LoRA modules, ISM retains only the dominant singular directions of each update via SVD truncation, scales them by a damping factor, and folds them into the deployed module. To handle the resulting feature-space shift, ISM applies an identity-anchored ridge-regression drift compensation to transport old-class statistics, followed by Local Classifier Alignment (LCA) to retrain the classification head. The authors provide a theoretical bound (Theorem 1) linking the merged model's loss increase to the weight step between consecutive modules and the classification head change. Experiments on four class-incremental benchmarks (CIFAR-100, CUB-200, ImageNet-R, Stanford Cars), two domain-incremental benchmarks, and robustness evaluations show that ISM achieves the best average final accuracy among compared rehearsal-free methods.

Critical strengths
  1. Well-motivated and practically relevant. The paper correctly identifies that most existing merging rules (Task Arithmetic, TIES, MagMax, etc.) were designed for one-shot merging of independently trained experts and are suboptimal when applied to sequential task streams. Designing a merging operator specifically for incremental deployment is a meaningful and timely contribution.

  2. Elegant and memory-efficient design. The combination of SVD truncation and a damping factor is clean: the damping factor controls the spectral step size (directly connected to Theorem 1), while truncation removes low-energy directions that may carry noise or interference. The entire pipeline operates in closed form with constant memory—only the current deployed module and the newly trained module are kept—and requires no stored data from past tasks.

  3. Comprehensive experimental evaluation. The paper compares against 17 rehearsal-free continual learning methods across four class-incremental benchmarks, two domain-incremental benchmarks, and robustness tests (CIFAR-100-C, CIFAR-100-P). The controlled merge-rule comparison in Appendix C (Table 6), where only the merging operator is swapped while all other stages remain fixed, is particularly well designed and informative. The ablation over truncation variants, energy fraction, and damping factor (Figure 2, Table 4) is thorough.

Critical weaknesses
  1. The theoretical analysis does not specifically justify spectral truncation. Theorem 1 bounds the loss increase by the weight step ‖W_{t+1} − W_t‖ and the classification head change. This explains why a small damping factor α is beneficial, but it does not explain why SVD truncation—keeping only the dominant singular directions—is preferable to simply scaling the full update matrix, applying L2 regularization during fine-tuning, or using other forms of spectral clipping. The paper acknowledges (Section 4.5) that "the spectral distance in Theorem 1 is set by the largest kept singular value and does not change with p," meaning the bound is insensitive to the truncation ratio. The actual benefit of truncation is attributed qualitatively to removing low-energy directions that affect "typical inputs rather than the worst case," but no formal analysis or empirical evidence (e.g., measuring the alignment between discarded directions and old-task feature subspaces) is provided to substantiate this claim. As a result, the theoretical contribution feels incomplete: it motivates damping but not the core spectral truncation mechanism.

  2. The drift compensation step raises scalability concerns. The ridge-regression map in Eq. (3) requires computing (F_{t−1}^T F_{t−1} + λI)^{−1}, which involves a d × d matrix inversion with O(d³) complexity. For ViT-B/16 (d = 768), this is tractable, but the cost becomes significant for larger backbones (e.g., ViT-L with d = 1024 or LLMs with d ≥ 4096). The paper does not discuss this computational bottleneck or explore alternatives such as low-rank approximations, iterative solvers (e.g., conjugate gradient), or the Woodbury identity that could reduce the cost to O(n²d) when n < d. Given that the paper positions ISM as a general-purpose continual learning framework, the absence of any scalability discussion is a notable gap.

  3. Missing comparison with strong recent baselines, and close methodological overlap. The paper does not compare against InfLoRA[1] or KeepLoRA[2] methods that enforce direction-level protection during continual learning. InfLoRA projects new-task updates onto the null space of old-task input covariance to guarantee zero interference, and KeepLoRA selectively freezes important LoRA directions—both are highly relevant and competitive baselines in the current continual learning landscape. InfLoRA is cited in the Related Work but absent from all experimental tables. Furthermore, ISM's post-hoc correction pipeline (drift compensation via a linear map followed by classifier realignment) is conceptually very close to the drift compensation and prototype correction strategies employed in these methods. The paper should (i) include experimental comparisons with InfLoRA under the same protocol, and (ii) provide a clearer discussion of what distinguishes ISM's approach from these existing direction-selection and post-hoc correction strategies beyond the SVD truncation operator itself.

  4. Evaluation is limited to class-incremental learning; generalizability to other scenarios is unclear. All four main benchmarks use the class-incremental setting with disjoint label sets and a growing classifier. While the paper includes two domain-incremental benchmarks (Office-Home, DomainNet), it does not evaluate on task-incremental learning (where task identity is available at test time), few-shot incremental learning, or language model continual learning. The reliance on per-class Gaussian statistics for drift compensation and LCA also assumes that each task provides sufficient data to estimate reliable class means and covariances. It remains unclear how ISM would perform when per-task data is very limited (n ≪ d), when class distributions are heavily imbalanced, or when the label space is fixed but the input distribution shifts continuously without discrete task boundaries. A discussion or preliminary experiment addressing at least one of these scenarios would strengthen the paper's claims of generality.

[1] InfLoRA: Interference-free low-rank adaptation for continual learning. CVPR, 2024.

[2] KeepLoRA: Continual Learning with Residual Gradient Adaptation. ICLR, 2026.

Questions for authors
  1. Could the authors provide an empirical analysis of what the discarded singular directions actually represent? For instance, measuring the cosine similarity or subspace overlap between the truncated (low-energy) directions of Δ and the feature subspaces of old tasks would directly support the claim that truncation removes interference rather than useful information.

  2. Table 5 shows that the optimal damping α varies considerably across datasets (0.10–0.30) and the ridge λ varies by an order of magnitude (10–100). In a deployment scenario without access to a validation set, how would a practitioner select these hyperparameters? Have the authors explored any adaptive or online strategy for setting α based on, e.g., the spectral norm of the update or the feature-space shift magnitude?

  3. How does ISM interact with the rank of the LoRA module? The default rank is r = 64. For very low ranks (e.g., r = 8 or 16), the number of singular values is small, and the energy truncation at p = 0.9 may retain nearly all directions, effectively disabling the truncation. Conversely, for very high ranks, the computational cost of SVD may increase. Could the authors comment on the sensitivity of ISM to the LoRA rank?

7 comments
AuthorsOpenReview ↗

Response to Reviewer q6ZZ (1/2)

We thank the reviewer for the detailed review. We would like to address each point below.

Weaknesses

[q6ZZ-W1]

Theorem 1 motivates damping but not truncation; no formal or empirical evidence that the discarded directions are interference rather than useful information.

Response. Both halves are addressed in the General response; in brief.

Theory. The new lemma (Section 3.5, Appendix A.1) states that, when only layer kk changes, E∥ft+1(x)−ft(x)∥≥μkκk∑iσk,i2\mathbb{E}\|f_{t+1}(x) - f_t(x)\| \ge \mu_k \kappa_k \sqrt{\sum_i \sigma_{k,i}^2}, with σk,i\sigma_{k,i} the singular values of the weight step, μk\mu_k a lower-Lipschitz constant of the frozen part after layer kk and κk\kappa_k a feature-excitation constant. Truncation removes the smallest σk,i\sigma_{k,i} and lowers this bound; the spectral-norm term of the upper bound is unchanged.

Evidence. Table 4 now holds p=1p = 1 at every damping value; the General response summarises it. Removing the tail changes nothing at α≤0.3\alpha \le 0.3 and, at α≥0.5\alpha \ge 0.5, lowers forgetting and raises the accuracy of the oldest tasks while the newest task is unchanged, the behaviour of interference rather than of task information. Appendix J measures it directly (see Q1).

[q6ZZ-W2]

The drift map needs a d×dd \times d inversion, O(d3)O(d^3); no discussion of scalability or of cheaper solvers.

Response. Added to the new Discussion section, with the measurements in Appendix L (see the General response). For d=768d = 768 the dense solve takes 0.03 s per task, below 0.1 % of the 88 s a task costs after fine-tuning. The solve is O(d3)O(d^3) only when n≥dn \ge d; for n<dn < d the Woodbury identity gives an n×nn \times n solve, O(n2d)O(n^2 d) (below 0.01 s for a few-shot task with n=100n = 100 or 200).

[q6ZZ-W3]

Missing comparison with InfLoRA and KeepLoRA; ISM's post-hoc pipeline is conceptually close to their drift compensation and prototype correction.

Response. We add KeepLoRA, which outperforms InfLoRA in its own comparison. KeepLoRA's paper reports no class-incremental ViT result, so we ran its official code in our pipeline with the same backbone, splits and seeds, its released recipe (rank 16, AdamW 10−310^{-3}, 10 epochs, ϵw=0.85\epsilon_w = 0.85, ϵf=0.99\epsilon_f = 0.99) and, in place of its CLIP text head, a linear classifier of the same kind as ours. Table 1, Last / Inc:

CIFAR-100CUB-200ImageNet-RStanford Cars
KeepLoRA88.70 / 92.8280.27 / 86.5578.61 / 83.5954.32 / 60.48
ISM (ours)91.95 / 94.8288.90 / 92.1683.68 / 87.5483.72 / 87.09

[q6ZZ-W4]

Evaluation limited to class-incremental learning; unclear behaviour with little data per task, imbalance, or no task boundaries.

Response. Table 2 covers domain-incremental learning with a fixed label space. Appendix H adds the low-data regime: KK training images per class and task on ImageNet-R, K∈{5,10}K \in \{5, 10\}, with E$^2$-LoRA and LCA in our pipeline and SLCA and EASE with their public implementations on the same splits; with n<dn < d the ridge solve uses the Woodbury form (Section 5). FA / AA:

KISME$^2$-LoRALCASLCAEASE
564.31 / 64.6553.46 / 54.3662.78 / 62.6356.91 / 59.4247.54 / 53.79
1071.84 / 75.6761.64 / 63.3469.92 / 73.9566.87 / 68.8354.65 / 60.65

ISM leads at both KK, by 1.5 and 1.9 points of final accuracy over LCA and by about 10 over E$^2$-LoRA. Task-incremental learning is easier than our setting; continuous shift without task boundaries needs further investigation, with more baselines to compare with.

AuthorsOpenReview ↗

Response to Reviewer q6ZZ (2/2)

Questions

[q6ZZ-Q1]

What do the discarded singular directions represent? Measure their overlap with the old-task feature subspaces.

Response. We measured this on ImageNet-R for every task from the second on and every adapted layer. The update Wt−Mt−1W_t - M_{t-1} is split by its SVD into the directions the truncation keeps and the directions it discards. We then pass 2,048 images of the old classes through the deployed model Mt−1M_{t-1}, collect the inputs of the layer, and take the subspace that holds 90 % of their energy as the old-task feature subspace of that layer. For each set of directions we report its share of the update's energy, its share of the change the update produces in the layer's output on the old-class inputs, and the fraction of its energy that lies inside the old-task subspace. At p=0.1p = 0.1, averaged over seeds, tasks and layers:

KeptDiscarded
Share of the update's energy0.170.83
Share of the output change on old-class inputs0.240.76
Overlap with the old-task feature subspace0.860.71

The discarded directions lie largely inside the old-task feature subspace and produce three quarters of the change the update causes on old-class inputs. They act on the features the old classes use, and removing them is what protects the old tasks at large steps in Table 4, where p=0.1p = 0.1 lowers forgetting by up to 1.0 point against p=1p = 1 with the newest task unchanged.

[q6ZZ-Q2]

The optimal α\alpha (0.10–0.30) and λ\lambda (10–100) vary across datasets. How would a practitioner choose them without a validation set? Any adaptive strategy?

Response. The loss from a shared value is small: on ImageNet-R, α∈{0.1,0.3,0.5}\alpha \in \{0.1, 0.3, 0.5\} are within 0.5 points of each other and every pp is within 0.25 of the best cell at α=0.3\alpha = 0.3 (Fig. 2). Appendix I tests two adaptive rules: a bound on the spectral norm of the deployed step, αtσ1(Δt)≤τ\alpha_t \sigma_1(\Delta_t) \le \tau, and a bound on the feature shift it causes, each with its bound fixed once; in the same runs λ\lambda is chosen per task by cross-validation on the task's own features. On ImageNet-R:

RuleRealised α\alphaλ\lambdaFA / AA
ISM (ours)0.3010083.48 / 87.59
Spectral norm, τ=0.1\tau = 0.10.33 (0.25 to 0.41)100083.28 / 87.55
Feature shift, κ=0.15\kappa = 0.150.35 (0.29 to 0.40)100083.26 / 87.52

Both rules stay within 0.2 points of ISM without any tuning, and the cross-validation picks λ=1000\lambda = 1000 at every task.

[q6ZZ-Q3]

How does ISM interact with the LoRA rank? At low rank the truncation may keep everything; at high rank the SVD costs more.

Response. Appendix F reports ranks 16, 32, 64 (default) and 128 on all four benchmarks. Final accuracy:

RankCIFAR-100CUB-200ImageNet-RStanford Cars
1690.3487.7779.7782.09
3291.6388.5482.6683.39
64 (paper)91.9588.9083.6883.72
12891.9989.1883.7283.45

Accuracy rises steeply from 16 to 32 and flattens from 64 on. At rank 64, p=0.9p = 0.9 keeps on average 34 of the 64 directions of a layer on ImageNet-R (23 at the first task, 40 at the last); at rank 16 it keeps 13 of 16, so the truncation is nearly inactive there, as the reviewer expects. The SVD acts on the r×dinr \times d_{\text{in}} factor, so its cost grows with rr: 0.03, 0.04, 0.11 and 0.52 s per task at ranks 16, 32, 64 and 128 (Table 11), far below the fine-tuning cost.

I thank the authors for their response, which has resolved most of my concerns. However, the effectiveness and efficiency of the proposed method on larger-scale models remain unclear, and the few-shot setting lacks comparison with dedicated few-shot continual learning methods, e.g [1]., . For these reasons, I can only support a score of 2.5 (marginal accept).

[1] Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models.

Reviewer_b28COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes a new method called Incremental Spectral Merging (ISM) for exemplar-free class-incremental learning (CIL) with a frozen pre-trained ViT and one LoRA module that is updated after each task. During the update we select, truncate, the leading singular directions that holds energy fraction p of the updates' energy. After such merging, the old-class Gaussian statistic for the classifiers is updated with a ride map as a drift compensation step. The authors evaluated the method on the standard split dataset CIL setup with 10 tasks ove four datasets, domain-incremental (office-home, DomainNet), and robustness eveluation with cifar-100-{c,p}.

Critical strengths
  1. Relevant setting of exemplar-free CIL - which still is the hardest setting for continual learning methods, here the work focuses on efficient classification with model merging inspired approach.
  2. The idea behind IMS is intuitive and simple.
  3. Paper is clearly written and understendable.
Critical weaknesses
  1. "keeping the dominant low-rank information is key to mitigating forgetting" is not so well supported. The gains seems to come mostly from existing components of the classifier (LCA) with a drift compensation (Tab.8) and the merging itself, not especially in the low-rank, do not change the results a lot (Tab.6, ie. IN-R /.Stanford Cars / CUB).
  2. Fairness and selection of hyper-parameters. Here, it's not clear how all the hyperparameters were selected for other methods. Based on the results from Fig.2 and Imagenet-R, the selection of the best hyper-paramerters for ISM was done on the test set itself. How it was done for the other methods? In the same way?
  3. A "constant memory" is a bit overstated here. Indeed for merging, yes. However, for every class we need to store covariance metrix (d^2, e.g.. 768 x 768). This storage is growing with the number of classes and the memory footprint of it will be bigger than LoRA module.
Questions for authors
  1. Why besalines OPCM and NUFILT mentioned in Spectral Merging section of related work are not present in the comparision?
  2. How hyper-parameters were selected for the other methods and comparison in Tab.6?
  3. How many singular directions are actaully kept at p = 0.9, for each layer and task? Can we do more insightful analysis of presenting what's happening with the spectra of W_t − M_{t−1} look like?
  4. Can we see the results for a longer sequences, e.g. 20 or 50 tasks?
  5. In DIL - are per-class statistics are updated similarly as in CIL or is something different?
  6. FA for CIFAR-100 in Tab.3 are weeker than for the corresponding methods in Tab.1. Why?
2 comments
AuthorsOpenReview ↗

Response to Reviewer b28C (1/2)

We thank the reviewer for the clear summary and for the questions on fairness and memory.

Weaknesses

[b28C-W1]

"keeping the dominant low-rank information is key to mitigating forgetting" is not so well supported; the gains seem to come mostly from the classifier (LCA) with drift compensation (Tab. 8), and the merge rule does not change the results a lot (Tab. 6).

Response. On the theory: the revision adds a lemma (Section 3.5, proof in Appendix A.1) that gives the missing argument for truncation. Theorem 1 bounds the worst-case change of the representation through σ1\sigma_1, which damping controls; the lemma bounds the typical change from below by the Frobenius norm of the kept update, ∑iσi2\sqrt{\sum_i \sigma_i^2}, which truncation lowers. Table 4 shows the effect: at α≥0.5\alpha \ge 0.5, keeping a tenth of the energy of each update beats keeping all of it (p=1p = 1) by 0.2 to 1.0 points, entirely through the old tasks, with the newest task unchanged. On the stage table (Table 8 of the submission, Table 9 of the revised PDF): the table is built stage by stage, so each row adds one stage on top of the previous ones, and the drift-compensation and alignment stages are measured on the merged backbone. They depend on it: the drift map is fitted between consecutive deployed modules, and the class statistics it transports are only usable when the feature space moves little between tasks, which is what the merge provides. This is why merging alone adds nothing on CUB-200 but the merge and the alignment together add 11.2 points, and why the Shapley share of merging is larger than its stand-alone gain on the fine-grained benchmarks.

[b28C-W2]

Fairness and selection of hyper-parameters: the best hyper-parameters for ISM were selected on the test set; how was it done for the other methods?

Response. Table 1 compares against the published numbers of each method. For Table 6, where we ran every merge rule ourselves, the submission tuned DARE, DELLA and KnOTS on ImageNet-R over a 2×22 \times 2 grid and reused the setting elsewhere, ran task arithmetic, TIES and max-abs at unit coefficient, and set ISM's α\alpha per dataset. The revised Table 6 removes this asymmetry: every method is tuned per dataset over four settings (Appendix C, Table 7). ISM stays first on the average, 87.00 against 86.68 for the best published rule, and on CIFAR-100, ImageNet-R and Stanford Cars. The settings tried (Table 7 of the revised PDF):

Merge ruleTuned parametersFour settings
ISMdamping α, energy fraction pα ∈ {0.1, 0.3} × p ∈ {0.7, 0.9}
Model Stocknoneone setting
Task Arithmeticcoefficient cc ∈ {0.3, 0.5, 0.7, 1.0}
DELLAcoefficient c, drop ratec ∈ {0.3, 1.0} × {0.5, 0.9}
KnOTS-TIEScoefficient c, kept rankc ∈ {0.3, 1.0} × {50, 100} %
DAREcoefficient c, drop ratec ∈ {0.3, 1.0} × {0.5, 0.9}
TIEScoefficient c, trimc ∈ {0.3, 1.0} × {20, 100} %
Max-abscoefficient cc ∈ {0.3, 0.5, 0.7, 1.0}
NUFILTcoefficient c, null-space rankc ∈ {0.1, 0.3} × {32, 64}

[b28C-W3]

"Constant memory" is overstated: a covariance matrix per class (768×768768 \times 768) grows with the number of classes and exceeds the LoRA module.

Response. We agree. The constant-memory property holds for the adapter and the merge inputs, not for the class statistics, which are 7682+768≈0.59768^2 + 768 \approx 0.59M numbers per class (59M on CIFAR-100, 118M on CUB-200 and ImageNet-R) against 8.26M parameters in the LoRA module. The new Discussion section gives this accounting, and Appendix K measures two cheaper statistics on ImageNet-R:

Class statisticsNumbers per classFA / AA
ISM (ours)590,59283.48 / 87.59
Mean and diagonal covariance1,53682.31 / 86.54
Mean alone76882.13 / 86.31

The diagonal covariance costs 1.2 points of final accuracy and the mean alone 1.4.

AuthorsOpenReview ↗

Response to Reviewer b28C (2/2)

Questions

[b28C-Q1]

Why are OPCM and NUFILT, mentioned in the related work, not in the comparison?

Response. NUFILT is now a row of Table 6, tuned per dataset over four settings like every other rule. Since NUFILT already outperforms OPCM in its own comparison, we include NUFILT as the representative of this line of work.

[b28C-Q2]

How were hyper-parameters selected for the other methods in Tab. 6?

Response. Answered under W2.

[b28C-Q3]

How many singular directions are kept at p=0.9p = 0.9, per layer and task? Can we see the spectra of Wt−Mt−1W_t - M_{t-1}?

Response. On ImageNet-R at rank 64, the number of directions kept at p=0.9p = 0.9, out of 64 per layer, averaged over the 12 blocks:

Task123456789
qkv172224282932333436
fc1222829333436373839
fc2293739444446464647

Over all layers and tasks the mean is 34, the minimum 6 and the maximum 56. In the spectrum of Wt−Mt−1W_t - M_{t-1} the leading direction holds 19 % of the energy and the top eight 58 %, averaged over layers and tasks.

[b28C-Q4]

Results for longer sequences, e.g. 20 or 50 tasks?

Response. Appendix G now reports 20- and 50-task splits of ImageNet-R and CIFAR-100 against the baselines of the E$^2$-LoRA paper (Table 12). Last / Inc of ISM and of the strongest baseline:

ImageNet-R 20 tasksImageNet-R 50 tasksCIFAR-100 20 tasksCIFAR-100 50 tasks
E$^2$-LoRA80.77 / 86.3778.58 / 83.9691.42 / 94.5590.70 / 93.86
ISM (ours)82.08 / 86.6878.64 / 84.2490.35 / 94.3687.05 / 92.63

ISM leads on ImageNet-R at both lengths, by 1.3 points of final accuracy at 20 tasks and level at 50, and trails E$^2$-LoRA on CIFAR-100 by 1.1 points at 20 tasks and 3.7 at 50 tasks.

[b28C-Q5]

In DIL, are per-class statistics updated similarly as in CIL?

Response. Yes, with one difference: the label space is shared, so we keep one mean and covariance per domain and class. After each domain the drift map is fitted on that domain's training features and applied to all stored statistics of earlier domains, and the alignment stage samples from every domain seen so far and trains the head over the shared label space.

[b28C-Q6]

FA for CIFAR-100 in Tab. 3 is weaker than for the same methods in Tab. 1. Why?

Response. Table 1 reports the numbers published by each method. Table 3 needs the models themselves to evaluate them on CIFAR-100-C/P.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We thank the reviewers for their careful reading. The three reviews converge on three requests: (i) a theoretical and empirical account of what spectral truncation adds over a damped step, (ii) a fair and fully disclosed comparison with other merge rules and with recent LoRA-based baselines, and (iii) an honest accounting of memory and compute. The revised PDF answers them as follows.

Theory. Section 3.5 adds a lemma (proof in Appendix A.1): under the assumptions of Theorem 1, the expected representation change is bounded from below by the Frobenius norm of the kept update, ∑iσi2\sqrt{\sum_i \sigma_i^2}. Truncation lowers this term; the upper bound of Theorem 1, set by σ1\sigma_1, is unchanged. Damping controls the worst case, truncation the typical case. The lemma bounds representation change, not forgetting; the head stages remain necessary.

Truncation versus damped interpolation. Table 4 now reports p=1p = 1 next to p=0.9p = 0.9 and p=0.1p = 0.1 at every damping value on ImageNet-R, from the sweep of Fig. 2. Two regimes appear. For α≤0.3\alpha \le 0.3 the energy fraction has no measurable effect (all pp within 0.2 points). For α≥0.5\alpha \ge 0.5 a smaller pp is better and the gap grows with α\alpha: p=0.1p = 0.1 beats p=1p = 1 by 0.2, 0.7 and 1.0 points at α=0.5,0.7,1\alpha = 0.5, 0.7, 1, entirely through the old tasks (forgetting 0.5 to 1.0 lower, oldest five tasks 0.4 to 1.3 higher) with the newest task unchanged. Appendix J measures where the discarded directions act: at p=0.1p = 0.1 they produce three quarters of the update's effect on old-class inputs and lie largely inside the old-task feature subspace.

Baselines and fairness. KeepLoRA is added to Table 1, run with its official code in our pipeline. Table 6 is rebuilt: every method is tuned per dataset over four settings (Appendix C, Table 7); NUFILT is added, and the table gains an average column. ISM stays first on the average and on three of the four benchmarks.

Memory and compute. "Constant memory" was too strong: the adapter and the merge inputs are constant, the per-class mean and covariance grow with the number of classes. The new Discussion section gives the storage accounting and the cost of the drift solve, with its Woodbury form for n<dn < d; Appendix K measures two cheaper class statistics, the mean with a diagonal covariance (1.2 points of final accuracy for 385 times less storage per class) and the mean alone (1.4 points for 769 times less), and Appendix L the cost of the drift solve, below 0.2 % of a task with either solver.

New appendices (same backbone, splits and three seeds as the paper): F, LoRA rank 16 to 128 with the cost of the merge per rank; G, 20- and 50-task sequences; H, few-shot tasks with 5 and 10 training images per class; I, two adaptive damping rules with cross-validated λ\lambda; J, what the discarded directions carry; K, cheaper class statistics; L, the cost of the drift solve; M, the drift map fitted on a replay buffer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.