- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes SCPO, a fine-tuning method for continual learning with model merging. Motivated by the observation that old-task loss is more sensitive to changes along the dominant singular directions of the pretrained weights, SCPO weakens gradient updates along these directions. Experiments on CLIP ViT models and Llama-3.2-1B show gains over standard fine-tuning with five merging methods.
Critical strengths
-
The method works as a plug-in for any merger. SCPO changes only the fine-tuning step and works with all five mergers tested. This makes it easy to adopt in existing merging pipelines.
-
The diagnostic experiments are well controlled. Same-rank random subspaces and norm-matched perturbations help rule out subspace size and update magnitude as explanations. This supports the claim that the direction of an update affects forgetting.
Critical weaknesses
-
Limited novelty. Protecting the pretrained principal subspace during continual fine-tuning closely resembles KeepLoRA [1]. The low-rank optimization builds directly on OSD [2], and the soft attenuation is similar to SGP [3].
-
Gains may come largely from the optimizer. In the ablations, much of the improvement over AdamW seems to come from switching to OSD, with core protection adding a modest margin (Tables 4 and E.4). The main tables and the LLM experiment lack an OSD-only baseline.
-
Key design choices are not fully supported. History weighting is not directly compared with a fixed pretrained core. At later stages, filtering task vectors after training performs close to SCPO (Table E.11).
References
[1] KeepLoRA: Continual Learning with Residual Gradient Adaptation. ICLR 2026.
[2] Memory-Efficient LLM Training with Online Subspace Descent. NeurIPS 2024.
[3] Continual Learning with Scaled Gradient Projection. AAAI 2023.
Questions for authors
- How much does core protection improve over OSD alone across all five mergers and on the LLM experiment?
- How does SCPO compare with a version that protects the fixed pretrained core without history weighting?
- With more seeds, does SCPO clearly outperform post-hoc filtering of OSD task vectors at later stages?
11 comments
Response to y8Wi - Weakness 1 about Differences from SGP
(2) Differences from SGP: SCPO uses two-sided protection derived from weight spectra, whereas SGP applies one-sided scaling in historical activation subspaces.
We acknowledge that SGP has already introduced soft projection to improve the stability–plasticity trade-off. Accordingly, soft attenuation itself is not the novelty we claim.
SGP constructs subspaces from the input activations of previous tasks, estimates the importance of each basis direction using the corresponding singular values, and scales gradients along these directions during subsequent tasks. As learning proceeds, it updates its feature bases and accumulated importance. Under the relevant weight-matrix storage convention, this operation takes the form of a projection on one side of the gradient.
SCPO, by contrast, uses both the left and right spectral information of the weight matrix. For a gradient , its protection operation is
Here, and act on the two sides of the matrix update, respectively, while subtracting avoids double-counting their overlap. Consequently, the protection scope of SCPO is not restricted to a one-sided feature subspace or to updates entirely within the core. It covers components involving either protected input directions or protected output directions.
To illustrate this, consider the case without history weighting, where and . The gradient can be decomposed as
G = {}& L_0GR_0 + L_0G(I-R_0) \\ & + (I-L_0)GR_0 + (I-L_0)G(I-R_0). \end{aligned}$$ These four components correspond to the core–core component, the two cross components between the core and its orthogonal complement, and the component in the orthogonal complements on both sides. SCPO multiplies the first three components by $\beta$ and leaves the fourth unchanged: $$\widetilde G = \beta \left[ L_0GR_0 + L_0G(I-R_0) + (I-L_0)GR_0 \right] + (I-L_0)G(I-R_0).$$ This decomposition clarifies the role of the two-sided design: it protects both the core–core component and the connections between the core and the remaining directions, while allowing the component in the orthogonal complements on both sides to continue adapting to new tasks. A one-sided projection generally covers only the components associated with one side, so the two protection operations are not equivalent. These properties hold for the gradient filtering operator. We do not extend them to a claim that the actual parameter displacement after the optimizer step, or the complete training trajectory, remains unchanged.Response to y8Wi - Weakness 1 about Relationship to OSD
We fully agree with the reviewer's assessment of OSD. The low-rank optimization component of SCPO builds directly on OSD. We will state this more clearly and will not present this component as an independent contribution of our work.
OSD provides the low-rank optimization mechanism, while SCPO additionally specifies which directions of the matrix update should be protected and how this protection should be adjusted according to the relationship between pretrained and historical weights. Using OSD alone does not automatically construct the two-sided, history-weighted core response operator described above. What must therefore be established is whether this additional design has a practical effect, rather than reattributing the gains from OSD to SCPO.
To this end, we added a controlled ablation that holds OSD fixed. In the ViT-B/16, eight-task setting with one random seed, the methods are initialized from the same historical weights at each stage and use the same data order, learning rate, and number of update steps. Only the choice between one-sided and two-sided protection is changed, and evaluation is performed before merging:
| Protection scheme | Mean new-task ACC over all eight stages ↑ | Mean degradation on previous tasks over stages 2–8 ↓ |
|---|---|---|
| One-sided soft protection | 91.1417% | 9.1794 pp |
| SCPO two-sided soft protection | 91.1544% | 8.1358 pp |
Degradation on previous tasks is defined as the mean accuracy on previous tasks before training at the current stage minus the mean accuracy on those tasks after training and before merging.
The new-task accuracies of two-sided and one-sided protection are similar, differing by 0.0127 percentage points. Meanwhile, two-sided protection reduces degradation on previous tasks by 1.0436 percentage points. This result supports the observation that two-sided protection can further reduce damage to previously acquired capabilities while maintaining similar new-task learning performance.
We will make clear that this experiment evaluates the role of the two-sided design and is not a comparison against the complete SGP or KeepLoRA algorithms. We also will not use a single-seed result to claim statistical significance or a universal advantage.
We are grateful to the reviewer for helping us define the scope of our contribution more precisely. The revised manuscript will fully acknowledge the relevant ideas in KeepLoRA, SGP, and OSD, and will describe the novelty of SCPO as follows: for continual model merging, SCPO combines a pretrained-core reference, weighting based on spectral overlap with historical weights, and two-sided soft protection into a concrete update mechanism. Its protection targets and operations differ from one-sided projection driven by historical activations and from constraints on the LoRA update subspace, and the practical role of its two-sided design is supported by controlled ablations.
Response to y8Wi - Weakness 1 about Differences from KeepLoRA
We sincerely thank the reviewer for drawing our attention to these closely related works. We agree that the original manuscript did not sufficiently explain the relationship between SCPO and existing methods, which could lead readers to view it as a straightforward combination of existing components. We apologize for this lack of clarity. In the revision, we will more explicitly acknowledge the contributions of prior work and explain the specific differences in what SCPO protects, how it uses historical information, and how it modifies updates.
We do not claim to be the first to introduce pretrained subspace protection, soft gradient attenuation, or low-rank optimization. The contribution of SCPO is a two-sided soft protection mechanism for continual model merging that uses pretrained weights as a reference and adjusts protection according to spectral overlap with historical weights. During new-task learning, it protects updates involving the input and output directions of the pretrained core while preserving adaptability in the remaining directions. We clarify its relationship to KeepLoRA, SGP, and OSD below.
(1) Differences from KeepLoRA: The source of protection information and the way updates are constrained differ.
KeepLoRA and SCPO both seek to preserve pretrained knowledge during continual adaptation, and we acknowledge this shared motivation. However, the protection operations they apply are different.
KeepLoRA combines dominant directions of the pretrained weights with feature directions from previous tasks to construct a subspace to protect. It constrains update directions by initializing one LoRA factor in the orthogonal complement of this subspace and keeping that factor fixed while optimizing the other factor during training on the current task. Thus, its protection mechanism is closely tied to the LoRA parameterization, and historical information is accumulated through task feature subspaces.
SCPO does not impose this constraint through a fixed LoRA factor, nor does it require extracting and accumulating activation subspaces from previous tasks. Instead, it obtains the left and right spectral subspaces of the pretrained weights and the merged weights from the preceding stage, and uses their overlap to construct protection weights. Let (L0,R0) denote the left and right core projectors of the pretrained weights, and let (Lt−1,Rt−1) denote the corresponding projectors of the historical merged weights. Then,
At=L0Lt−1L0,Bt=R0Rt−1R0.
SCPO therefore retains the pretrained core as its reference while adjusting protection strength according to the alignment between the historical merged model and this core. These history-dependent weights characterize the geometric overlap between spectral subspaces; we do not equate them with a proven measure of task importance.
Furthermore, KeepLoRA restricts the allowable update directions through its parameterization, whereas SCPO applies soft protection to gradients. When the protection coefficient β>0, the protected directions retain a limited capacity for adaptation. The distinction therefore extends beyond using different subspace sizes: it concerns the source of historical information, the constraints on parameter updates, and the construction of protection strength.