acceptodds
Desk rejected at ICLR 2027

Fine-Tune to Merge: Spectral Core Preservation for Continual Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning aims to enable large pretrained models to acquire new knowledge while retaining previously learned capabilities, yet remains challenged by catastrophic forgetting during sequential adaptation. Existing model-merging-based approaches mitigate forgetting by integrating model updates across tasks, but most still rely on standard fine-tuning and largely overlook the resulting perturbations to the pretrained core subspace. We find that old-task loss is more sensitive to parameter perturbations along pretrained core directions than along residual directions, making new-task updates along these directions more likely to interfere with previously learned knowledge. To address this issue, we propose Spectral Core-Preserving Optimization (SCPO), a fine-tuning method that protects the pretrained core subspace when learning new tasks and produces task updates better suited to model merging-based continual learning. SCPO defines the protection scope using the dominant singular subspaces of the pretrained weights and adaptively modulates the protection strength according to their overlap with the corresponding subspaces of the previous merged model, thereby selectively attenuating gradient components along protected directions. The filtered gradients are further optimized within an online-updated low-dimensional subspace, and the resulting task vectors are subsequently integrated through model merging. Extensive experiments on vision models and large language models demonstrate the effectiveness of SCPO. Across five representative model merging methods, SCPO consistently improves over standard fine-tuning, yielding an average relative accuracy gain of 5.44% and a maximum gain of 8.63%.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, model merging, fine-tuning, spectral subspace, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y8WiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SCPO, a fine-tuning method for continual learning with model merging. Motivated by the observation that old-task loss is more sensitive to changes along the dominant singular directions of the pretrained weights, SCPO weakens gradient updates along these directions. Experiments on CLIP ViT models and Llama-3.2-1B show gains over standard fine-tuning with five merging methods.

Critical strengths
  1. The method works as a plug-in for any merger. SCPO changes only the fine-tuning step and works with all five mergers tested. This makes it easy to adopt in existing merging pipelines.

  2. The diagnostic experiments are well controlled. Same-rank random subspaces and norm-matched perturbations help rule out subspace size and update magnitude as explanations. This supports the claim that the direction of an update affects forgetting.

Critical weaknesses
  1. Limited novelty. Protecting the pretrained principal subspace during continual fine-tuning closely resembles KeepLoRA [1]. The low-rank optimization builds directly on OSD [2], and the soft attenuation is similar to SGP [3].

  2. Gains may come largely from the optimizer. In the ablations, much of the improvement over AdamW seems to come from switching to OSD, with core protection adding a modest margin (Tables 4 and E.4). The main tables and the LLM experiment lack an OSD-only baseline.

  3. Key design choices are not fully supported. History weighting is not directly compared with a fixed pretrained core. At later stages, filtering task vectors after training performs close to SCPO (Table E.11).


References

[1] KeepLoRA: Continual Learning with Residual Gradient Adaptation. ICLR 2026.

[2] Memory-Efficient LLM Training with Online Subspace Descent. NeurIPS 2024.

[3] Continual Learning with Scaled Gradient Projection. AAAI 2023.

Questions for authors
  1. How much does core protection improve over OSD alone across all five mergers and on the LLM experiment?
  2. How does SCPO compare with a version that protects the fixed pretrained core without history weighting?
  3. With more seeds, does SCPO clearly outperform post-hoc filtering of OSD task vectors at later stages?
11 comments
AuthorsOpenReview ↗

Response to y8Wi - Weakness 1 about Differences from KeepLoRA

We sincerely thank the reviewer for drawing our attention to these closely related works. We agree that the original manuscript did not sufficiently explain the relationship between SCPO and existing methods, which could lead readers to view it as a straightforward combination of existing components. We apologize for this lack of clarity. In the revision, we will more explicitly acknowledge the contributions of prior work and explain the specific differences in what SCPO protects, how it uses historical information, and how it modifies updates.

We do not claim to be the first to introduce pretrained subspace protection, soft gradient attenuation, or low-rank optimization. The contribution of SCPO is a two-sided soft protection mechanism for continual model merging that uses pretrained weights as a reference and adjusts protection according to spectral overlap with historical weights. During new-task learning, it protects updates involving the input and output directions of the pretrained core while preserving adaptability in the remaining directions. We clarify its relationship to KeepLoRA, SGP, and OSD below.

(1) Differences from KeepLoRA: The source of protection information and the way updates are constrained differ.

KeepLoRA and SCPO both seek to preserve pretrained knowledge during continual adaptation, and we acknowledge this shared motivation. However, the protection operations they apply are different.

KeepLoRA combines dominant directions of the pretrained weights with feature directions from previous tasks to construct a subspace to protect. It constrains update directions by initializing one LoRA factor in the orthogonal complement of this subspace and keeping that factor fixed while optimizing the other factor during training on the current task. Thus, its protection mechanism is closely tied to the LoRA parameterization, and historical information is accumulated through task feature subspaces.

SCPO does not impose this constraint through a fixed LoRA factor, nor does it require extracting and accumulating activation subspaces from previous tasks. Instead, it obtains the left and right spectral subspaces of the pretrained weights and the merged weights from the preceding stage, and uses their overlap to construct protection weights. Let (L0,R0)(L_0, R_0) denote the left and right core projectors of the pretrained weights, and let (Lt−1,Rt−1)(L_{t-1}, R_{t-1}) denote the corresponding projectors of the historical merged weights. Then,

At=L0Lt−1L0,Bt=R0Rt−1R0.A_t = L_0 L_{t-1} L_0, \qquad B_t = R_0 R_{t-1} R_0.

SCPO therefore retains the pretrained core as its reference while adjusting protection strength according to the alignment between the historical merged model and this core. These history-dependent weights characterize the geometric overlap between spectral subspaces; we do not equate them with a proven measure of task importance.

Furthermore, KeepLoRA restricts the allowable update directions through its parameterization, whereas SCPO applies soft protection to gradients. When the protection coefficient β>0\beta > 0, the protected directions retain a limited capacity for adaptation. The distinction therefore extends beyond using different subspace sizes: it concerns the source of historical information, the constraints on parameter updates, and the construction of protection strength.

AuthorsOpenReview ↗

Response to y8Wi - Weakness 1 about Differences from SGP

(2) Differences from SGP: SCPO uses two-sided protection derived from weight spectra, whereas SGP applies one-sided scaling in historical activation subspaces.

We acknowledge that SGP has already introduced soft projection to improve the stability–plasticity trade-off. Accordingly, soft attenuation itself is not the novelty we claim.

SGP constructs subspaces from the input activations of previous tasks, estimates the importance of each basis direction using the corresponding singular values, and scales gradients along these directions during subsequent tasks. As learning proceeds, it updates its feature bases and accumulated importance. Under the relevant weight-matrix storage convention, this operation takes the form of a projection on one side of the gradient.

SCPO, by contrast, uses both the left and right spectral information of the weight matrix. For a gradient GG, its protection operation is

G~=G−(1−β)(AtG+GBt−AtGBt).\widetilde G = G - (1-\beta) \left(A_tG + GB_t - A_tGB_t\right).

Here, AtGA_tG and GBtGB_t act on the two sides of the matrix update, respectively, while subtracting AtGBtA_tGB_t avoids double-counting their overlap. Consequently, the protection scope of SCPO is not restricted to a one-sided feature subspace or to updates entirely within the core. It covers components involving either protected input directions or protected output directions.

To illustrate this, consider the case without history weighting, where At=L0A_t=L_0 and Bt=R0B_t=R_0. The gradient can be decomposed as

G = {}& L_0GR_0 + L_0G(I-R_0) \\ & + (I-L_0)GR_0 + (I-L_0)G(I-R_0). \end{aligned}$$ These four components correspond to the core–core component, the two cross components between the core and its orthogonal complement, and the component in the orthogonal complements on both sides. SCPO multiplies the first three components by $\beta$ and leaves the fourth unchanged: $$\widetilde G = \beta \left[ L_0GR_0 + L_0G(I-R_0) + (I-L_0)GR_0 \right] + (I-L_0)G(I-R_0).$$ This decomposition clarifies the role of the two-sided design: it protects both the core–core component and the connections between the core and the remaining directions, while allowing the component in the orthogonal complements on both sides to continue adapting to new tasks. A one-sided projection generally covers only the components associated with one side, so the two protection operations are not equivalent. These properties hold for the gradient filtering operator. We do not extend them to a claim that the actual parameter displacement after the optimizer step, or the complete training trajectory, remains unchanged.
AuthorsOpenReview ↗

Response to y8Wi - Weakness 1 about Relationship to OSD

We fully agree with the reviewer's assessment of OSD. The low-rank optimization component of SCPO builds directly on OSD. We will state this more clearly and will not present this component as an independent contribution of our work.

OSD provides the low-rank optimization mechanism, while SCPO additionally specifies which directions of the matrix update should be protected and how this protection should be adjusted according to the relationship between pretrained and historical weights. Using OSD alone does not automatically construct the two-sided, history-weighted core response operator described above. What must therefore be established is whether this additional design has a practical effect, rather than reattributing the gains from OSD to SCPO.

To this end, we added a controlled ablation that holds OSD fixed. In the ViT-B/16, eight-task setting with one random seed, the methods are initialized from the same historical weights at each stage and use the same data order, learning rate, and number of update steps. Only the choice between one-sided and two-sided protection is changed, and evaluation is performed before merging:

Protection schemeMean new-task ACC over all eight stages ↑Mean degradation on previous tasks over stages 2–8 ↓
One-sided soft protection91.1417%9.1794 pp
SCPO two-sided soft protection91.1544%8.1358 pp

Degradation on previous tasks is defined as the mean accuracy on previous tasks before training at the current stage minus the mean accuracy on those tasks after training and before merging.

The new-task accuracies of two-sided and one-sided protection are similar, differing by 0.0127 percentage points. Meanwhile, two-sided protection reduces degradation on previous tasks by 1.0436 percentage points. This result supports the observation that two-sided protection can further reduce damage to previously acquired capabilities while maintaining similar new-task learning performance.

We will make clear that this experiment evaluates the role of the two-sided design and is not a comparison against the complete SGP or KeepLoRA algorithms. We also will not use a single-seed result to claim statistical significance or a universal advantage.

We are grateful to the reviewer for helping us define the scope of our contribution more precisely. The revised manuscript will fully acknowledge the relevant ideas in KeepLoRA, SGP, and OSD, and will describe the novelty of SCPO as follows: for continual model merging, SCPO combines a pretrained-core reference, weighting based on spectral overlap with historical weights, and two-sided soft protection into a concrete update mechanism. Its protection targets and operations differ from one-sided projection driven by historical activations and from constraints on the LoRA update subspace, and the practical role of its two-sided design is supported by controlled ablations.

Reviewer_DmFEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies continual learning with model merging, where a pretrained model is fine-tuned sequentially on each new task from the previous merged model and the task updates are merged incrementally. Motivated by the observation that old-task loss is more sensitive to perturbations along the pretrained core subspace than along its complement, the authors propose Spectral Core-Preserving Optimization (SCPO), which modifies only the fine-tuning stage. For each weight matrix, SCPO measures the subspace overlap between the pretrained core and the top-k singular subspaces of the previous merged model, uses it to attenuate core-related gradient components, and optimizes the filtered gradients with Online Subspace Descent (OSD).

Critical strengths
  1. The method is clear.

  2. SCPO modifies only the fine-tuning stage, making it compatible with existing merging methods.

Critical weaknesses
  1. Contributions of the core design components, namely core protection and the merging scheme. Under the controlled setting in Table E.11, the gains over OSD at stages 4 and 8 are under 0.5 points, and at these stages OSD with post-hoc filtering nearly matches SCPO, suggesting that the order of the core protection step matters little. Similarly, in Table 3, standard fine-tuning with Iso-C is only 1.0 point below SCPO + Iso-C, weakening the significance of core protection.

  2. There are many inconsistencies in the reported results. For example, SCPO + TA on 8-task ViT-B/32 is reported as 84.3, 84.4, 84.42, and 83.918 in Tables 1, 3, 4, and E.4. The authors need to explain these inconsistencies. The headline claim is also misleading. The 5.44% average relative gain is measured against AdamW and thus includes the contribution of OSD.

Questions for authors
  1. Is the experimental setting task-incremental, with the task ID provided during evaluation?

  2. How were the fixed checkpoints and merging coefficients in Table E.4 obtained?

  3. Could the authors clarify the definition of CP-AdamW? Section 4.5 describes it as fixed pretrained core protection, while Table D.3 and Appendix E.2.1 describe a dynamic historical core.

  4. What is the contribution of the history weighting? An ablation with A_t = L_0 and B_t = R_0 would isolate it from fixed pretrained core protection.

  5. How sensitive is SCPO to the hyperparameters β and k, and how were their values chosen?

  6. What are the training budgets of tangent-space fine-tuning and SCPO in Table E.5? Since the budgets differ, are the two methods directly comparable?

6 comments
AuthorsOpenReview ↗

Response to DmFE - Major Comment 1/2

Major Comment 1

We sincerely thank the reviewer for the careful comments on contribution attribution and the timing of protection. We understand and agree with your concern: improvements in overall performance and accuracy comparisons at only a few stages are not sufficient to establish that training-time protection is indispensable. Our original manuscript did not express this conclusion with sufficient caution, and we sincerely apologize for this.

We would like to distinguish two related but separate questions: whether core protection makes an independent contribution, and whether training-time protection consistently outperforms post-hoc filtering.

First, to disentangle the contributions of core protection and the subsequent merging procedure, we have added shared-history experiments. At each stage, all methods start from the same checkpoint, with the OSD optimizer, data order, learning rate schedule, and number of update steps held fixed, and are evaluated before merging. Here, CC denotes the component whose input and output directions both lie within the pretrained core subspaces. Full SCPO additionally applies history-weighted protection to the two types of cross directions for which only one side lies within a core subspace.

ViT-B/32, eight tasksMean new-task ACC ↑Mean degradation on previous tasks ↓
OSD88.8635%4.7713 pp
Protection of the left–right core intersection (CC) only88.9352%4.7000 pp
SCPO88.9787%3.7745 pp

We appreciate the reviewer's reminder that our original conclusion was too strong, and we will temper our statements regarding the “necessity” of training-time protection. Likewise, for the approximately one-percentage-point improvement with Iso-C, we will not claim that core protection is indispensable based on the magnitude of the gain alone. Instead, we will use the controlled experiments above to explain its incremental contribution. Thank you again for helping us distinguish more clearly the respective contributions of core protection, the timing of protection, and the merging procedure.

Major Comment 2

We sincerely thank the reviewer for carefully checking the values across different tables and for raising the issue of performance attribution. We understand that these insufficiently explained differences can affect the clarity and credibility of the results. The original manuscript did not adequately specify the experimental sources, evaluation splits, and aggregation procedures, and we sincerely apologize for these omissions.

Upon re-examination, we found that these differences involve distinct aggregation procedures, reporting precision, and evaluation splits; they cannot all be attributed to rounding:

  • 84.4 in Table 3 and 84.42 in Table 4 correspond to the same validation-set result of 84.4242% with seed 43. The difference is due to the number of decimal places reported.
  • For 84.3 in Table 1, the available records for five seeds yield a recomputed mean of 84.3178%, which is consistent with the reported mean. We will further verify how the standard deviation in the original table was computed and standardize the reporting.
  • 83.918 in Table E.4 corresponds to an independent test-set result of 83.9176%, obtained on a different evaluation split from the validation results above.

We acknowledge that the original manuscript did not clearly identify these distinctions and that its general description of the evaluation protocol was inconsistent with the actual sources of some table entries. This is a reporting issue that requires correction. In the revision, we will verify the source of the results in each table and explicitly specify the data split, random seeds, aggregation procedure, and reporting precision, so that validation and test results are not presented together without clear distinction.

Regarding the 5.44% average relative improvement, we agree with the reviewer that this value represents the gain of the complete SCPO method over standard AdamW fine-tuning. It includes contributions from both OSD and core protection and therefore cannot be attributed entirely to the newly introduced soft core attenuation mechanism. We thank the reviewer again for identifying these issues. We will prioritize correcting the reporting conventions and contribution attribution so that each conclusion is supported by clear, verifiable experimental evidence.

AuthorsOpenReview ↗

Response to DmFE - Question 1/2/3

Question 1

We sincerely thank the reviewer for identifying this aspect of the experimental setting that requires further clarification. Yes, our vision experiments use a task-incremental setting in which task identity is available at evaluation time.

Specifically, all tasks share a single visual encoder that is continually updated. When evaluating a task, its known identity determines the corresponding fixed classification head and class set. Thus, task identity is used to select the classification head; it is not used to select or invoke different historical encoders or task-specific expert models. All compared methods follow the same evaluation protocol.

We sincerely apologize for not making this sufficiently explicit in the original manuscript. We will clearly state this assumption in the experimental setup of the revised manuscript to avoid any interpretation of these results as class-incremental evaluation, where prediction is performed jointly over the classes of all tasks without access to task identity.

Question 2

We sincerely thank the reviewer for this question. The phrase “fixed checkpoints and merging coefficients” in the original manuscript was overly brief and could cause misunderstanding. We apologize for this lack of clarity.

Table E.4 uses the deployment checkpoints saved at each stage of the existing continual training runs. These checkpoints were obtained as follows:

  1. Task-model selection. We fine-tune on the current task within the prescribed training budget and select the fine-tuned checkpoint based on its performance on the current task's validation set.
  2. Merging-coefficient selection. We select the merging coefficient using the equally weighted mean validation accuracy across all tasks seen so far. The candidate range is 0.1–3.0 in increments of 0.1 for TA, and 0.5–3.0 in increments of 0.1 for Iso-C. The search stops when three consecutive candidates fail to improve the validation performance.
  3. Saving the deployment model. We merge using the selected coefficient and save the deployment checkpoint for that stage, which is then used for initialization on subsequent tasks and for evaluation.
  4. Independent testing. To produce Table E.4, we freeze these checkpoints and their previously selected merging coefficients and evaluate them only on the independent test sets. Test results are not used to reselect checkpoints, adjust coefficients, or retrain models.

Accordingly, “fixed” means that the models and coefficients already selected along each method's training trajectory remain unchanged during independent test evaluation. It does not mean that all methods and stages use the same numerical merging coefficient, nor that different methods share the same historical model.

We will include this procedure in the revised manuscript and explicitly distinguish model and coefficient selection on the validation sets from final reporting on the test sets. We appreciate the reviewer's comment, which helps us describe the experimental protocol more precisely for reproducibility.

Question 3

We sincerely thank the reviewer for carefully identifying this inconsistency. Your observation is correct: the description of CP-AdamW as using fixed pretrained core protection in Section 4.5 is a writing error on our part. The actual experiments in that section use dynamic historical core protection, consistent with Table D.3 and Appendix E.2.1. We sincerely apologize for the confusion this error has caused in understanding the method and interpreting the experiments.

Specifically, at the beginning of each task, this CP-AdamW variant extracts the leading left and right singular subspaces from the weights of the previously merged model, Wt−1W_{t-1}, and uses them to construct the core protection operator for the current task. The reference core therefore changes across task stages, rather than always being derived from the initial pretrained model, W0W_0. Here, “dynamic” refers to updating the reference core across task stages, not recomputing the core subspaces at every optimization step.

We thank the reviewer again for bringing this issue to our attention. We will carefully correct this description and check the consistency of the related terminology throughout the manuscript.

AuthorsOpenReview ↗

Response to DmFE - Question 4/5

Question 4

We sincerely thank the reviewer for this specific and helpful ablation suggestion. We agree that directly comparing history weighting with fixed pretrained core protection is necessary to identify the contribution of history weighting more clearly. We apologize for the absence of this comparison in the original manuscript. Following your suggestion, we have added a fixed-core variant by setting At=L0A_t = L_0 and Bt=R0B_t = R_0, and compared it with SCPO using history weighting. The experiments use ViT-B/32, eight tasks, and Task Arithmetic, with k=16k = 16 and β=0.02\beta = 0.02. All other training and evaluation settings are matched, and we use three paired random seeds. The results on the full test sets are reported below as means and sample standard deviations:

Core protection mechanismFinal ACC ↑BWT ↑
Fixed pretrained core83.6868 ± 0.0409%−5.0568 ± 0.2998 pp
History-weighted core84.0226 ± 0.0700%−4.5788 ± 0.4290 pp
Difference in means+0.3358 pp+0.4781 pp

Question 5

We sincerely thank the reviewer for pointing out the shortcomings in our hyperparameter descriptions and sensitivity analysis. We agree that clarifying the roles of these two parameters, the rationale for their selected values, and their effects on performance is essential for assessing the reliability of the method. Specifically, kk controls the dimensionality of the pretrained core subspaces, while β\beta controls the extent to which core-related gradients are retained. A smaller β\beta applies stronger attenuation to directions with higher core responses, whereas β=1\beta = 1 disables this protection operation. For the history-weighted variant, β=0\beta = 0 does not imply that all core-related gradient components are completely zeroed out. To assess sensitivity, we have added the complete 4×64 \times 6 grid under the ViT-B/32, eight-task, Task Arithmetic setting. The ACC values on the full test sets with seed 43 are shown below:

k\βk\backslash\beta00.010.020.030.050.1
883.420483.652483.925183.542384.003383.6979
1683.829783.698084.057484.400183.863483.7589
3283.614684.065483.977583.778083.739583.5019
6484.029584.102583.631684.354883.933683.5385

Across the 24 configurations, ACC ranges from 83.4204% to 84.4001%, a total spread of 0.9797 percentage points. With the default β=0.02\beta = 0.02 held fixed, varying kk produces a spread of 0.4258 percentage points. These results indicate that, within the range evaluated here, performance does not depend on a particularly narrow region of high-performing hyperparameter values.

We will also clearly distinguish the original procedure for determining the default values from this additional sensitivity analysis: the test-set grid is used to assess the sensitivity of the existing default configuration, not to reselect the default parameters. Since these experiments use only one random seed, we will not interpret them as evidence of stability across seeds. We appreciate the reviewer's reminder and will include the complete results and an explanation of parameter selection in the revised manuscript.

Reviewer_JNNiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

This paper investigates the impact of parameter perturbations on previously learned tasks in the context of continual learning with model merging. The authors empirically observe that updates along the dominant singular subspaces (the "pretrained core") of the weights cause more significant degradation in old-task performance compared to residual directions. Building on this observation, they propose Spectral Core-Preserving Optimization (SCPO), a fine-tuning method that selectively attenuates gradient components along these core directions based on their spectral overlap with the previously merged model. The filtered gradients are subsequently optimized using Online Subspace Descent (OSD). Extensive experiments across three vision backbones (ViT-B/32, ViT-B/16, ViT-L/14) and a seven-task language benchmark (Llama) demonstrate that SCPO consistently improves average accuracy (ACC) and backward transfer (BWT) when integrated with various merging strategies.

Critical strengths

Strengths

  1. Well-Motivated and Elegant Design: The paper is well-motivated by the solid empirical observation that old-task loss is highly sensitive to perturbations in the pretrained core subspace. Integrating spectral core protection directly into the optimization phase, without altering the subsequent merging rules, makes SCPO highly modular and compatible with existing merging techniques.
  2. Extensive and Comprehensive Evaluation: The empirical validation is rigorous and broad. It covers three different vision backbones, task sequences of varying lengths (8, 14, and 20 tasks), and a language benchmark. The results provide strong evidence of SCPO's effectiveness and scalability beyond the vision domain. Notably, the combination of SCPO and Iso-C achieves the best ACC and BWT across all nine configurations in Table 1.
Critical weaknesses

Weaknesses

Major Comments:

  1. Marginal Contribution of Soft Core Attenuation over OSD: A significant concern is that the performance gains of SCPO appear to be largely driven by the underlying OSD optimizer rather than the proposed soft core attenuation mechanism itself. For instance, in Table 4, simply replacing AdamW with OSD yields a substantial ACC improvement (from 79.22% to 83.30%), while adding SCPO only provides a marginal additional gain of 1.12%. Similarly, under the Iso-C merging rule (Table E.4), the improvement of SCPO over OSD is merely ~0.38 points. This raises doubts about the actual contribution and necessity of the core preservation mechanism when OSD is already employed.
  2. Limited Evaluation of Plasticity: The current evaluation of SCPO’s plasticity (ability to learn new tasks) is insufficient. Table E.10 only reports the new-task accuracy at three arbitrarily selected stages. To rigorously verify whether the reduction in catastrophic forgetting comes at the expense of plasticity, the authors should report the pre-merge accuracy for each stage, ensuring that AdamW, OSD, and SCPO are initialized from identical historical weights and trained under strictly identical computational budgets.
  3. Lack of Justification and Sensitivity Analysis for Hyperparameter kk: The protection rank kk is fixed at 16 (as shown in Table D.4). The rationale behind this specific choice is not discussed. Furthermore, the sensitivity of SCPO's performance to this critical hyperparameter remains entirely unclear.

Minor Comments:

  1. Missing Baseline in Main Table: The results for the vanilla continual Iso-C (C. Iso-C) baseline are missing from Table 1. This omission makes it difficult for readers to make a direct and fair comparison between the vanilla C. Iso-C and the proposed SCPO + Iso-C.
  2. Missing Discussion on Recent Related Works: The related work section should be updated to discuss and distinguish SCPO from recent relevant literature that also leverages singular subspaces for model merging and continual learning. Specifically:
    • Spectral Rewiring for Exploration, Purification, and Model Merging (arXiv:2607.03065), which aligns task updates with the subspace constructed by the top singular vectors of pretrained weights to improve capability preservation.
    • DC-Merge: Improving Model Merging with Directional Consistency (CVPR 2026), which projects task vectors into a shared subspace constructed from their top singular vectors.
Questions for authors

Questions to Authors

  1. Contribution Isolation: Given that OSD already provides a significant performance boost, how do you isolate and justify the specific contribution of the soft core attenuation mechanism? Could you provide further theoretical insights or more fine-grained ablation studies to prove that the core protection is not redundant when combined with OSD?
  2. Missing Baselines: Could you please include the results for the vanilla continual Iso-C in Table 1 to allow a direct comparison with SCPO + Iso-C?
  3. Related Works Distinctions: How does SCPO fundamentally differ from Spectral Rewiring (arXiv:2607.03065) and DC-Merge (CVPR 2026), given that both also utilize the top singular vectors of weights to construct subspaces for preserving capabilities or reducing interference? Please clarify these distinctions in the revised manuscript.
8 comments
AuthorsOpenReview ↗

Response to JNNi - Major Comment 1

Major Comment 1

We sincerely thank the reviewer for the careful comments regarding the attribution of our method's contributions. We fully appreciate and agree with this concern: OSD itself contributes important performance improvements, whereas the original manuscript did not sufficiently distinguish the role of the underlying optimizer from the incremental contribution of the proposed core protection mechanism. We sincerely apologize for this lack of clarity and will revise the relevant statements to avoid attributing the entire improvement over AdamW to core protection.

To address this concern more directly, we have added controlled experiments. At each learning stage, OSD and SCPO start from exactly the same historical checkpoint and use the same training data order, learning rate schedule, and number of update steps. We evaluate new-task learning performance and degradation on previous tasks before model merging. Here, degradation on previous tasks is defined as the decrease in their mean accuracy from before to after training; a smaller value indicates better retention of previously acquired capabilities.

ModelMethodMean new-task accuracy ↑Mean degradation on previous tasks ↓
ViT-B/32OSD88.8635%4.7713 pp
ViT-B/32SCPO88.9787%3.7745 pp
ViT-B/16OSD91.2339%9.3820 pp
ViT-B/16SCPO91.1544%8.1358 pp

All results above are evaluated on the full test sets. New-task accuracy is averaged over all eight stages, while degradation on previous tasks is averaged over stages 2–8, where previous tasks are present.

For ViT-B/32, adding core protection reduces mean degradation on previous tasks by 0.9968 percentage points, or approximately 20.9%, while increasing mean new-task accuracy by 0.1152 percentage points. In particular, reduced degradation is observed at all seven stages involving previous tasks. For ViT-B/16, mean degradation on previous tasks decreases by 1.2462 percentage points, or approximately 13.3%, while mean new-task accuracy decreases by 0.0795 percentage points.

We thank the reviewer again for raising this concern and helping us describe the contribution of each component more accurately.

AuthorsOpenReview ↗

Response to JNNi - Major Comment 2

Major Comment 2

We sincerely thank the reviewer for this specific and constructive experimental suggestion. We agree that reporting new-task accuracy at only three stages is insufficient to comprehensively evaluate plasticity or rule out the effects of different historical initializations or training budgets. The experimental description and evidence in the original manuscript were indeed insufficient in this respect, and we sincerely apologize for this shortcoming.

Following your suggestion, we have added a controlled evaluation across all eight stages. At each stage, AdamW, OSD, and SCPO are initialized from the same historical checkpoint, with the set of trainable parameters, data splits and order, learning rate schedule, batch size, number of training epochs, and number of update steps matched across methods. We evaluate the models at the end of the fixed training budget and record new-task accuracy before any model merging operation.

The results on the full test sets for ViT-B/32 are as follows:

MethodStage 1: CarsStage 2: DTDStage 3: EuroSATStage 4: GTSRBStage 5: MNISTStage 6: RESISC45Stage 7: SUN397Stage 8: SVHNEight-stage mean
AdamW77.975473.563899.074199.129199.630095.650873.822197.238089.5104
OSD75.301673.297999.037098.733299.620095.127072.710897.080588.8635
SCPO75.848873.829898.963098.574899.620095.238172.720997.034488.9787

Compared with OSD, SCPO improves mean new-task accuracy by 0.1152 percentage points, while reducing mean degradation on previous tasks from 4.7713 to 3.7745 percentage points. This indicates that, under this controlled setting, improved retention of previous capabilities is achieved without a decrease in mean new-task accuracy.

We have also added the corresponding experiments for ViT-B/16. The eight-stage mean new-task accuracies of OSD and SCPO are 91.2339% and 91.1544%, respectively, with the latter being 0.0795 percentage points lower. Meanwhile, SCPO reduces mean degradation on previous tasks from 9.3820 to 8.1358 percentage points. We will report these results in full. We are grateful for the reviewer's suggestion, which has helped us evaluate this trade-off in a more comprehensive and verifiable manner.

AuthorsOpenReview ↗

Response to JNNi - Major Comment 3

Major Comment 3

We sincerely thank the reviewer for pointing out this omission. We agree that providing only the default protection rank, without sufficiently explaining its role or the sensitivity of performance to it, limits readers' ability to assess the reliability of the method. We sincerely apologize for this omission and have added a systematic hyperparameter analysis.

The protection rank kk determines the dimensionality of the dominant singular subspace of the pretrained weights and thus the range of directions subject to protection. To directly assess its sensitivity, we first fix β=0.02\beta=0.02 and evaluate k∈{8,16,32,64}k\in\{8,16,32,64\} in the ViT-B/32, eight-task setting with Task Arithmetic. The results on the full test sets for a single random seed are as follows:

Protection rank kk816 (default)3264
Test ACC (%) ↑83.925184.057483.977583.6316

The accuracy range across these four settings is 0.4258 percentage points. We have further added a complete 24-configuration grid combining these four values of kk with six values of β∈{0,0.01,0.02,0.03,0.05,0.1}\beta\in\{0,0.01,0.02,0.03,0.05,0.1\}. Across all configurations, test ACC ranges from 83.4204% to 84.4001%, spanning 0.9797 percentage points.

These results support the observation that, within the range and setting examined here, performance does not depend on a narrowly tuned parameter interval, and k=16k=16 is one of several well-performing default configurations. We nevertheless acknowledge that these results do not establish k=16k=16 as optimal across different models and task sequences, nor do they replace an explanation of the original hyperparameter selection procedure.

We will include the complete sensitivity results in the revised manuscript and clearly distinguish the original basis for choosing the default values from the retrospective validation provided by these additional experiments. We thank the reviewer again for helping us explain this design choice and its limitations more thoroughly.

Protection rank kk / gradient-retention coefficient β\beta00.010.020.030.050.1
883.420483.652483.925183.542384.003383.6979
1683.829783.698084.057484.400183.863483.7589
3283.614684.065483.977583.778083.739583.5019
6484.029584.102583.631684.354883.933683.5385

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.