acceptodds
Under review as a conference paper at ICLR 2027

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

27 views

Abstract

Large language models (LLMs) acquire new skills through fine-tuning, but fine-tuning can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent work connects forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid the principal directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. We introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting the top 99% of singular directions on both sides of each weight matrix still leaves enough low-energy capacity to learn new tasks, without task-specific subspace estimation. Our experiments show that SST improves in-domain performance while retaining prior capabilities. This behavior scales with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing - less KL drift than the baseline recipes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Learning, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wJyAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Subspace Tuning (SST) to mitigate catastrophic forgetting during LLM fine-tuning. SST computes optimizer updates from full gradients, then projects them onto the non-principal singular subspaces of weight matrices using a two-sided projection. It optionally refreshes the protected subspaces to balance adaptation and retention.

Critical strengths

Potential benefits for capability retention. In Table 5, no-refresh SST achieves similar average in-domain accuracy to SFT, with better retention of prior capabilities. These results suggest that SST may offer a useful adaptation–retention trade-off in the evaluated settings.

Critical weaknesses
  1. Limited methodological novelty. OPLoRA already uses the same two-sided projection, and Adam-NSCL already projects Adam-generated updates. Compared with OPLoRA, SST optimizes full weights instead of LoRA factors and optionally refreshes the subspace. The paper does not clearly establish why these changes constitute a substantial methodological advance.

  2. The benefit of basis refresh is not well explained. The paper motivates refresh through subspace rotation, yet exact two-sided updates preserve the protected singular directions. If these directions remain the leading ones, recomputing the SVD should leave the projector unchanged. The authors should show how the protected subspace changes during training and explain how these changes produce the reported gains.

Questions for authors
  1. What is SST’s main technical contribution beyond OPLoRA’s two-sided projection? In the matched-protection comparison with OPLoRA, what adapter rank was used, and could limited adapter capacity explain the remaining performance gap?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer wJyA

W1 Limited methodological novelty. OPLoRA already uses the same two-sided projection, and Adam-NSCL already projects Adam-generated updates. Compared with OPLoRA, SST optimizes full weights instead of LoRA factors and optionally refreshes the subspace. The paper does not clearly establish why these changes constitute a substantial methodological advance.

R1 We thank the reviewer for raising the differences between SST and prior art. SST combines weight-derived spectral protection with full-gradient optimizer statistics, projection of the actual dense update, and an optional refresh mechanism, with broad empirical evidence for the learning–retention trade-off.

OPLoRA optimizes low-rank factors; Adam-NSCL protects previous-task activation subspaces. SST applies optimizer-update projection directly to full weights using subspaces derived from the pretrained weights, without requiring previous-task examples. These differences might seem minor but we present very strong empirical evidence and thorough analysis leading to real gains.

SST’s practical benefit is visible across domains: matched OPLoRA averages 74.8 / 73.1, versus SST(R100’s) 76.7 / 79.2, which improves both metrics in every domain as presented in Table 5. SST scales substantially better than prior art (such as OPLoRA) as shown in Figure 15 in the Appendix.

Upon the feedback of the reviewer, we also provide an updated FinQA ablation that supports the SST design: at identical 76.3 ID, SST retains 75.21, versus 54.58 when both gradients and updates are projected.

Table C. FinQA projection ablation.

MethodNo refresh: ID / retentionR100: ID / retention
SFT72.3 / 23.89—
Project gradient only77.8 / 71.1677.6 / 71.08
Project gradient and update76.3 / 54.5878.1 / 55.62
SST: full gradient, projected update76.3 / 75.2176.0 / 75.48

We have also added scaling experiments on Medicine and Pathology too in the Rebuttal Section C in the Appendix of the draft. We can clearly see that OPLoRA starts to collapse as the data scales. We hope that our in depth comparison with prior art as presented in Appendix B.9 as well as the new ablations clearly demonstrates the contribution of SST.

W2 The benefit of basis refresh is not well explained. The paper motivates refresh through subspace rotation, yet exact two-sided updates preserve the protected singular directions. If these directions remain the leading ones, recomputing the SVD should leave the projector unchanged. The authors should show how the protected subspace changes during training and explain how these changes produce the reported gains.

R2 We agree with the reviewer’s conditional statement: if the protected directions remain leading and the cutoff is separated, exact refresh leaves the projector unchanged.

However, preserving those directions does not preserve their ranking. Singular values in the free subspace can grow past the protected cutoff. Refresh then protects the new leading set and releases previously protected directions for adaptation. This changes the allowed update space, while keeping its dimension fixed, and provides a mechanism for changing the adaptation–retention balance.

We measured these changes through training steps.

For an orthonormal protected basis UU, its projector is P=UU⊤P = UU^\top. The distance between the old and refreshed projectors is

D = \left\\|P_{\mathrm{new}} - P_{\mathrm{old}}\right\\|_F = \sqrt{\sum_i \sum_j \left[(P_{\mathrm{new}})_{ij} - (P_{\mathrm{old}})_{ij}\right]^2}.

If one unit direction uu is replaced by an orthogonal unit direction vv, with the other directions unchanged, then Pnew−Pold=vv⊤−uu⊤P_{\mathrm{new}} - P_{\mathrm{old}} = vv^\top - uu^\top. Each direction's projector has squared Frobenius norm 1, because ∣uu⊤∣F2=(u⊤u)2=1\\|uu^\top\\|_F^2 = (u^\top u)^2 = 1 for a unit vector, and similarly for vv. Their cross-term is zero because u⊤v=0u^\top v = 0. Therefore,

D2=∣uu⊤∣F2+∣vv⊤∣F2−2(u⊤v)2=1+1−0=2,D^2 = \\|uu^\top\\|_F^2 + \\|vv^\top\\|_F^2 - 2(u^\top v)^2 = 1 + 1 - 0 = 2,

D=2≈1.414.D = \sqrt{2} \approx 1.414.

A value near 1.414 represents the same total subspace change as one orthogonal direction replacement. D = 0 means an unchanged span; D > 1.414 indicates significant change in the protected basis

Refresh updateMatrices with D > 1.414
100130/180 (72.2%)
200130/180 (72.2%)
300123/180 (68.3%)
400108/180 (60.0%)
50091/180 (50.6%)
60037/180 (20.6%)

Thus, refresh changes the constraint during training, especially early in this run. By protecting the current leading directions and releasing earlier ones, it allows subsequent adaptation to follow the updated spectrum.

Q1 What is SST’s main technical contribution beyond OPLoRA’s two-sided projection? In the matched-protection comparison with OPLoRA, what adapter rank was used, and could limited adapter capacity explain the remaining performance gap?

R1 SST computes optimizer statistics from full weight gradients, then constrains the actual dense update, with optional basis refresh. OPLoRA instead optimizes factor-coordinate gradients and states; it is not equivalent to projecting gradients and updates in direct weight space. We provide precise theoretical and empirical evidence supporting our claims in Section 4.1 and Appendix B.9; where OPLoRA performs substantially worse at matched subspace settings (r = 0.99d) as well as a larger protection setting (r=128).

MethodProtected rank kAdapter r / αScienceFinQAPathologyMedicineMean
OPLoRA, official setting12832 / 3278.5 / 79.172.8 / 61.777.4 / 51.670.0 / 60.074.7 / 63.1
OPLoRA, matched protection⌊0.99d⌋32 / 3280.8 / 79.271.5 / 72.376.2 / 70.770.8 / 70.074.8 / 73.1
SST/R100⌊0.99d⌋Dense; no adapter81.7 / 80.073.1 / 80.177.6 / 78.574.3 / 78.376.7 / 79.2

These results along with the results provided in R1 further improves the comparison of SST with OPLoRA at scale.

We are committed to addressing all of your concerns and have made our best effort to respond to each of them thoroughly. We would be more than happy to provide further clarification or address any remaining concerns. We would greatly appreciate it if you could consider our efforts and kindly reconsider your review rating in light of our responses.

Thank you for the response. However, my main concern remains. SST mainly combines existing ideas from OPLoRA and Adam-NSCL, and the rebuttal does not show that this combination is a substantial methodological contribution. I will therefore keep my score.

Reviewer_m3HCOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SST uses the top left and right singular vectors of pretrained weight matrices to apply a two-sided orthogonal projection to optimizer updates. The singular bases can be refreshed periodically to trade retention for adaptation. Experiments on the Qwen3 family cover Science, FinQA, Pathology, and Medicine, evaluating forgetting of general capabilities, model scale, optimizers, sequential tasks, model merging, protected rank, and computational cost. SST generally learns the target task while keeping losses in prior capabilities small.

Critical strengths

The projection point is technically well motivated. Adam's coordinate-wise preconditioning does not commute with subspace projection. The controlled study shows that projecting the raw gradient still leaves 69% of the applied update in protected directions, whereas projecting the final optimizer update reduces this leakage to 0%.

Critical weaknesses

The core mechanism is not identifiable from the current experiments. The paper does not compare against matched-dimensional random subspaces, bottom singular subspaces, or update-norm-matched controls. The protected-rank ablation simultaneously changes subspace identity, dimensionality, and effective update scale, so it does not establish that protecting principal singular directions is the source of the gain.

Questions for authors
  1. Under matched free dimensionality, per-step Frobenius update norm, and refresh frequency, how do random left/right subspaces and bottom singular subspaces compare with SST?
  2. If learning rate, training duration, or early stopping is tuned separately for SFT, SDFT, and SST, does SST retain its Pareto advantage when target-task accuracy or cumulative update norm is matched? Please report multi-seed means and standard deviations for at least Science and FinQA.
  3. What training-set size, number of epochs, optimizer-step count, random seeds, and checkpoint-selection rule were used for each task?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer m3HC

W1 Q1 The core mechanism is not identifiable from the current experiments. The paper does not compare against matched-dimensional random subspaces, bottom singular subspaces, or update-norm-matched controls. The protected-rank ablation simultaneously changes subspace identity, dimensionality, and effective update scale, so it does not establish that protecting principal singular directions is the source of the gain. Under matched free dimensionality, per-step Frobenius update norm, and refresh frequency, how do random left/right subspaces and bottom singular subspaces compare with SST?

R1 We add complete FinQA comparisons that vary subspace identity, projection point, and update scale under one common training/evaluation recipe. We compare top, random, and bottom subspaces at equal protected dimensions and refresh cadence, plus a dense update control matched to SST’s per-step update norm. Table F reports the completed results.

Table F. FinQA geometry and update-scale baselines; Each cell is ID / retention (%).

Methodno refreshR100
SST: top singular protection76.3 / 75.2176.0 / 75.48
Random protection76.5 / 74.0676.5 / 73.29
Bottom singular protection77.0 / 67.6278.0 / 67.98
SFT, norm matched to SST77.4 / 61.2478.0 / 62.96

SST retains more than random and bottom protection; compared with the dense norm control, SST gains 13.97 / 12.52 retention points, with 1.1 / 2.0 lower ID. This shows that matching update magnitude alone does not reproduce SST’s trade-off. We agree that the original rank ablation does not isolate these factors and we have updated the draft with these results in Appendix C.

Q2. If learning rate, training duration, or early stopping is tuned separately for SFT, SDFT, and SST, does SST retain its Pareto advantage when target-task accuracy or cumulative update norm is matched? Please report multi-seed means and standard deviations for at least Science and FinQA.

R2 We provide a learning rate sweep covering {1e-4, 5e-5, 1e-5} and scheduler {flat, cosine}, with the same training budget and length for SFT and SST below. We have added this ablation in Appendix C of the updated draft and present the results below. Establishing the optimal learning rate for each dataset and recipe is non-trivial, hence we rely on the sensitivity to optimization hyperparameters. SST is substantially less sensitive to these choices.

Learning rateSchedulerSFTSST, no refreshSST/R10
1e-4Flat45.3 / 3.274.7 / 73.876.7 / 63.9
5e-5Flat58.3 / 10.072.9 / 80.276.0 / 76.8
1e-5Flat75.5 / 62.569.9 / 81.171.8 / 80.5
1e-4Cosine67.2 / 5.175.2 / 78.676.5 / 70.1
5e-5Cosine72.2 / 24.173.6 / 80.176.2 / 78.1
1e-5Cosine78.0 / 66.370.4 / 81.370.0 / 81.7

To understand the early-stopping frontier, Figure 3 and the table below reports Science accuracy after every epoch: SST+R100 exceeds SFT in both ID and retention at all five checkpoints, including SFT’s best early-stopping point.

Table B. Performance on Science after every epoch.

EpochCumulative examplesSFT: ID / retentionSST/R100: ID / retention
19,00065.4 / 54.877.0 / 78.2
218,00064.5 / 52.875.9 / 79.0
327,00068.3 / 49.378.1 / 79.8
436,00070.9 / 50.479.1 / 78.8
545,00068.5 / 43.075.9 / 79.1

Additionally, Figure 3 compares 45,000 distinct examples over one epoch, with evaluations at 0.2-epoch intervals. In Figure 3, even the earliest SST/R100 checkpoint (77.0 / 78.2) exceeds SFT’s best displayed ID checkpoint (70.9 / 50.4). These results establish the advantage across the evaluated training durations. Figure 15 reports every epoch for Medicine/Pathology (9,000 examples × five epochs) and FinQA (2,000 × three epochs, versus 6,000 × one epoch); all these studies show that SST outperforms base recipes across hyperparameters and early stopping budgets.

Q3 What training-set size, number of epochs, optimizer-step count, random seeds, and checkpoint-selection rule were used for each task?

R3 We provide details in the training setup and hyperparameters in Appendix B.14. We use a cosine schedule, 10 warmup steps, Adam β=(0.9,0.999), ε=1e-8, weight decay 0 and a gradient clipping 1. The paper’s scaling studies report Science/Pathology/Medicine at 9,000 examples × 5 epochs, and FinQA at 2,000 × 3 epochs vs. 6,000 × 1 epoch; Science also includes 45,000 × 1 epoch. Evaluations occur after every epoch. SST leads the pareto throughout training epochs.

TaskTraining examplesEpochsOptimizer updates
Science9,00021,126
FinQA6,0002626
Pathology8,99921,126
Medicine8,99621,126

We are committed to addressing all of your concerns and have made our best effort to respond to each of them thoroughly. We would be more than happy to provide further clarification or address any remaining concerns. We would greatly appreciate it if you could consider our efforts and kindly reconsider your review rating in light of our responses.

Reviewer_6mVjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the forgetting of general capabilities of a pretrained model during downstream fine-tuning, and proposes Subspace Tuning (SST). The method constructs a protected subspace from the principal left and right singular directions of the pretrained weights, projects the actual weight update produced by the optimizer bilaterally onto the orthogonal complement, and can periodically recompute the protected basis. The paper distinguishes gradient projection from optimizer-update projection, emphasizing that Adam preconditioning can make the two produce different actual updates. Validation covers multiple domains, model scales, non-reasoning models, short-horizon continual fine-tuning and expert merging, and provides comparisons with close methods, protection ratios, refresh frequencies and computational overhead. The main results show that SST substantially mitigates the degradation of some general capabilities while preserving downstream task performance.

Critical strengths
  1. The main experiments show a practically meaningful learning-retention gain. In Table 4 (PDF p. 22), the SFT downstream-performance/old-capability averages on Science are 77.6/55.5, while SST without refresh gives 80.1/80.0; the corresponding figures for FinQA are 72.3/23.7 versus 73.0/82.3. These results show that the improvement is not obtained merely by reducing the amount of new-task learning. The paper also covers model scale and multiple application settings, which is the main experimental evidence supporting acceptance.
  2. Imposing the constraint on the optimizer's actual update has clear technical value. Section 2.2 points out that Adam's per-coordinate preconditioning generally does not commute with subspace projection, so constraining the raw gradient does not guarantee that the actual update stays inside the allowed subspace. The method is straightforward to implement, and the comparisons against Sculpting, OPLoRA, protection-ratio matching and KL regularization provide a useful basis for judging the practical role of this design.
Critical weaknesses
  1. The evaluation controls of key comparisons are inconsistent, which invalidates core attributions such as "KL regularization does not prevent forgetting". The base model in Table 4 (PDF p. 22) has MMLU-Pro/old-capability averages of 73.0/81.0, whereas in Table 6 (PDF p. 26) it has 59.7/78.0; the two sets of numbers cannot both describe the same baseline, yet the paper does not explain the differences in checkpoint, prompt, decoding setup or evaluation version. The comparison against the KL family in Table 6 therefore cannot directly support the conclusion that "KL regularization does not prevent forgetting". In addition, the only default learning rate is 5×10−55\times10^{-5} in Appendix B.12, while SFT's learning rate, training length and early-stopping frontier are not given; the near-zero HumanEval of some SFT runs in Table 4 may also come from the output format or generation length rather than from capability loss. Without a learning-forgetting frontier measured at matched ID performance, the size of SST's advantage over SFT cannot be determined accurately.
  2. The core conclusions cannot be checked in the main text, and the mechanistic evidence and the claimed scope exceed what the material supports. The main body (pp. 1–10) contains no tables at all: the only table carrying the main results is Table 4 in Appendix p. 22, and the caption of Figure 5 on p. 6 uses that table directly as its data source. Readers must jump between the main text and the appendix to verify the key comparison of SST 80.1/80.0 against SFT 77.6/55.5, and at least a condensed version of the main table should be moved into the main text. Moreover, the mechanistic probes in Figure 4 and Appendix B.7 use only 8 training examples and 4 generic sentences, which is not enough to generally support the claim that the optimizer state of projected gradients must be "mis-scaled", or that OPLoRA is equivalent to scheme (3) in direct weight space (the protection-ratio-matched OPLoRA already reaches ID 80.8/LM 79.2); the "rotate away" explanation of refreshing in Section 2.3 is also inconsistent with the analytic property of exact bilateral projection under a fixed protected basis. Finally, the total time overhead of R10 is 37.6% (p. 28) and the old-capability averages on Pathology and Medicine still drop by 8.7 and 9.8 points relative to the base model (p. 22), so the claims of "nearly eliminating forgetting" and of a "practical default" need to be narrowed by model scale, refresh frequency and task-sequence length.
Questions for authors
  1. Why do the base-model scores in Table 4 and Table 6 differ? Please specify the checkpoint, prompt, decoding, evaluation data and tool versions used in each, and report the existing SST/SFT/KL comparisons under the same configuration.
  2. In periodic refreshing, how often and by how much does the protected projection actually change? Can you show whether the changes mainly come from spectral crossing, degeneracy or numerical error, and revise the "rotate away" explanation accordingly?
  3. With the same protection ratio and training budget, and with the learning rate and training length separately tuned, does the SST learning-forgetting frontier still stably dominate OPLoRA, gradient projection and SFT? Please prioritize repeated runs on real tasks rather than relying only on small-scale mechanistic probes.
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 6mVj

W1.1 & Q1 The evaluation controls of key comparisons are inconsistent, which invalidates core attributions such as "KL regularization does not prevent forgetting". The base model in Table 4 (PDF p. 22) has MMLU-Pro/old-capability averages of 73.0/81.0, whereas in Table 6 (PDF p. 26) it has 59.7/78.0; the two sets of numbers cannot both describe the same baseline, yet the paper does not explain the differences in checkpoint, prompt, decoding setup or evaluation version. The comparison against the KL family in Table 6 therefore cannot directly support the conclusion that "KL regularization does not prevent forgetting".

R1.1 We thank the reviewer for pointing out this minor typo in the Appendix. The apparent inconsistency comes from a typo in Table 6’s base-model. The Qwen3-8B baseline is consistently reported as 81.0 Prior Capability Avg. across Tables 2, 3, 4 and 5; as well as the Figure 1 and 5 in the paper. Table 6 reports the same as 78.0 which we have rectified in the main draft.

However, we believe that the core claim - “KL regularization does not prevent forgetting” stays intact irrespective. Correcting the baseline leaves the central comparison unchanged: on Science, SST (R100) and KL with λ=0.001 both reach 81.7 ID, but retain 80.0 versus 62.5 - a 17.5 pp difference. On FinQA, SST retains 82.3 vs. 23.2 at similar ID (73.0 vs. 73.4).

We use Qwen/Qwen3-8B as the base model across our experiments; we use the official Qwen3 chat prompts with thinking enabled. The following table reports per dataset decode setting:

EvaluationSamplingMaximum new tokensPrompt / data
IDGreedy, T=0, top-p=132,768Task question in the Qwen3 chat template
GPQA-DiamondT=1, top-p=.95, top-k=5032,768198 questions; shuffled options; final ANSWER: $LETTER
MMLU-ProT=1, top-p=.95, top-k=5032,76812,032 test questions; reasoning, then The answer is (X)
GSM8KT=1, top-p=.95, top-k=508,1921,319 test questions; reasoning and boxed answer
HumanEvalT=1, top-p=.95, top-k=508,192164 functions; signature/docstring and code-only instruction
IFEvalT=1, top-p=.95, top-k=504,096541 original instruction prompts; prompt-level strict scoring

We will also be providing the precise reproduction code and evaluation scripts upon acceptance.

W1.2 In addition, the only default learning rate is (5 × 10⁻⁵) in Appendix B.12, while SFT's learning rate, training length and early-stopping frontier are not given;

R1.2 We additionally provide a learning rate sweep covering {1e-4, 5e-5, 1e-5} and scheduler {flat, cosine}, with the same training budget and length for SFT and SST below. We have added the corresponding plot in Appendix C of the main draft. Establishing the optimal learning rate for each dataset and recipe is non-trivial due to the huge compute cost, hence we rely on the sensitivity to optimization hyperparameters. SST is substantially less sensitive to these choices as shown in the table below.

Learning rateSchedulerSFTSST, no refreshSST/R10
1e-4Flat45.3 / 3.274.7 / 73.876.7 / 63.9
5e-5Flat58.3 / 10.072.9 / 80.276.0 / 76.8
1e-5Flat75.5 / 62.569.9 / 81.171.8 / 80.5
1e-4Cosine67.2 / 5.175.2 / 78.676.5 / 70.1
5e-5Cosine72.2 / 24.173.6 / 80.176.2 / 78.1
1e-5Cosine78.0 / 66.370.4 / 81.370.0 / 81.7

To understand the early-stopping frontier, Figure 3 and the table below reports Science accuracy after every epoch: SST+R100 exceeds SFT in both ID and retention at all five checkpoints, including SFT’s best early-stopping point.

Table B. Performance on Science after every epoch.

EpochCumulative examplesSFT: ID / retentionSST/R100: ID / retention
19,00065.4 / 54.877.0 / 78.2
218,00064.5 / 52.875.9 / 79.0
327,00068.3 / 49.378.1 / 79.8
436,00070.9 / 50.479.1 / 78.8
545,00068.5 / 43.075.9 / 79.1

Additionally, Figure 3 compares 45,000 distinct examples over one epoch, with evaluations at 0.2-epoch intervals. In Figure 3, even the earliest SST/R100 checkpoint (77.0 / 78.2) exceeds SFT’s best displayed ID checkpoint (70.9 / 50.4). These results establish the advantage across the evaluated training durations. Figure 15 reports every epoch for Medicine/Pathology (9,000 examples × five epochs) and FinQA (2,000 × three epochs, versus 6,000 × one epoch); all these studies show that SST outperforms base recipes across hyperparameters and early stopping budgets.

W1.3 the near-zero HumanEval of some SFT runs in Table 4 may also come from the output format or generation length rather than from capability loss.

R1.3 SFT scores 30.49 on HumanEval, versus the base model’s 90.85, and only 4/164 SFT outputs hit the 8,192-token limit. Even crediting all four yields only 32.93. HumanEval is a coding dataset and under the same prompt and extractor, failures to produce executable, correct code are genuine losses of the evaluated coding/instruction-following capability. These results show that the low HumanEval does not come from output format or generation length but is genuine capability loss.

W1.4 Without a learning-forgetting frontier measured at matched ID performance, the size of SST's advantage over SFT cannot be determined accurately.

R1.4 We believe that the new hyperparameter sweep and the tables in the Appendix provide a thorough pareto at matched or even better ID performance. We would like to provide any added ablation that can further solidify SST.

W2.1 The core conclusions cannot be checked in the main text, and the mechanistic evidence and the claimed scope exceed what the material supports. The main body (pp. 1–10) contains no tables at all: the only table carrying the main results is Table 4 in Appendix p. 22, and the caption of Figure 5 on p. 6 uses that table directly as its data source. Readers must jump between the main text and the appendix to verify the key comparison of SST 80.1/80.0 against SFT 77.6/55.5, and at least a condensed version of the main table should be moved into the main text.

R2.1 We thank the reviewer for pointing out the lack of Tables in the main draft. Our attempt was to provide an easier understanding of the core contributions and focus on the scaling trends of SST across parameters and data. We have added a condensed version of the main table to the main text from the Appendix.

W2.2 Moreover, the mechanistic probes in Figure 4 and Appendix B.7 use only 8 training examples and 4 generic sentences, which is not enough to generally support the claim that the optimizer state of projected gradients must be "mis-scaled",

R3.1 We provide results on FinQA expanding the mechanistic probe to a large scale experiment:

Table C. FinQA projection ablation.

MethodNo refresh: ID / retentionR100: ID / retention
SFT72.3 / 23.89—
Project gradient only77.8 / 71.1677.6 / 71.08
Project gradient and update76.3 / 54.5878.1 / 55.62
SST: full gradient, projected update76.3 / 75.2176.0 / 75.48

SST and gradient-plus-update projection both reach 76.3 ID, but SST retains 75.21 versus 54.58. The results above support the claim that the optimizer’s final update should be used to project instead of the raw gradients and projecting the update is the optimal choice for SST. We have added these results to the Appendix C.

W2.3 Q3 OPLoRA is equivalent to scheme (3) in direct weight space (the protection-ratio-matched OPLoRA already reaches ID 80.8/LM 79.2);

R2.3 OPLoRA optimizes LoRA factors, so its gradients and Adam statistics differ from direct weight-space projection. The cited near-match of OPLoRA and SST is on the Science dataset only. Across all four domains, matched-protection OPLoRA averages 74.8 / 73.1, versus SST(R100) 76.7 / 79.2; in-domain / prior respectively. SST(R100) improves both in-domain performance and retention on prior capability substantially. We have also added additional data scaling experiments comparing OPLoRA and SST in Appendix C of the udpated draft. OPLoRA forgets more as data scales, whereas SST stays intact. In summary, the comparable numbers for OPLoRA are only true for Science and in general it is suboptimal than SST.

W2.4 Q2 The "rotate away" explanation of refreshing in Section 2.3 is also inconsistent with the analytic property of exact bilateral projection under a fixed protected basis. In periodic refreshing, how often and by how much does the protected projection actually change? Can you show whether the changes mainly come from spectral crossing, degeneracy or numerical error, and revise the "rotate away" explanation accordingly?

R2.4 Exact bilateral updates preserve the currently protected singular directions, but do not guarantee that they remain leading. Free singular values can overtake the protected cutoff, changing the leading set without changing the frozen directions.

We leverage a regular refresh mechanism to adapt to a changing basis. We measured the change in the protected left projector at every refresh during training. For an orthonormal protected basis UU, its projector is P=UU⊤P = UU^\top. The distance between the old and refreshed projectors is

D = \left\\|P_{\mathrm{new}} - P_{\mathrm{old}}\right\\|_F = \sqrt{\sum_i \sum_j \left[(P_{\mathrm{new}})_{ij} - (P_{\mathrm{old}})_{ij}\right]^2}.

If one unit direction uu is replaced by an orthogonal unit direction vv, with the other directions unchanged, then Pnew−Pold=vv⊤−uu⊤P_{\mathrm{new}} - P_{\mathrm{old}} = vv^\top - uu^\top. Each direction's projector has squared Frobenius norm 1, because ∣uu⊤∣F2=(u⊤u)2=1\\|uu^\top\\|_F^2 = (u^\top u)^2 = 1 for a unit vector, and similarly for vv. Their cross-term is zero because u⊤v=0u^\top v = 0. Therefore,

D2=∣uu⊤∣F2+∣vv⊤∣F2−2(u⊤v)2=1+1−0=2,D^2 = \\|uu^\top\\|_F^2 + \\|vv^\top\\|_F^2 - 2(u^\top v)^2 = 1 + 1 - 0 = 2,

D=2≈1.414.D = \sqrt{2} \approx 1.414.

A value near 1.414 represents the same total subspace change as one orthogonal direction replacement. D = 0 means an unchanged span; D > 1.414 indicates significant change in the protected basis

Refresh updateMatrices with D > 1.414
100130/180 (72.2%)
200130/180 (72.2%)
300123/180 (68.3%)
400108/180 (60.0%)
50091/180 (50.6%)
60037/180 (20.6%)

As seen in the Table above, substantial changes are frequent early in training and less frequent later.

Refresh changes which directions are protected and releases others for subsequent adaptation throughout training. If leading membership remains unchanged and the cutoff is separated, exact refresh leaves the projector unchanged. However, we obsesrve changes in the projector space throughout training.

W2.5 Finally, the total time overhead of R10 is 37.6% (p. 28) and the old-capability averages on Pathology and Medicine still drop by 8.7 and 9.8 points relative to the base model (p. 22), so the claims of "nearly eliminating forgetting" and of a "practical default" need to be narrowed by model scale, refresh frequency and task-sequence length.

R2.5 The 37.6% overhead applies to the R10 version of SST. With R100, a 5.1% total overhead is observed and substantial in-domain and prior capability retention over unconstrained training; no-refresh projection costs 1.5%. The cited results of 8.7 and 9.8 points relative to the base model (p. 22) are the R10 variants of SST. With R100, retention is 78.5 / 78.3 on Pathology/Medicine, only 2.5 / 2.7 below the base, versus SFT’s 34.7 / 36.3. The trade-off is substantially better: Medicine improves on both axes; Pathology gains 43.8 retention pp at a 2.8 pp ID cost. That said, we understand your concern of making the claim more clear and we have updated the draft accordingly.

Comments on the paper

1 comment

We thank all the reviewers for their feedback. Our current revision focuses on learning rate sweep, extended baseline comparison and clearer explanations of SST’s design.

We have added a dedicated Rebuttal (Appendix C) section containing the additional experiments. All changes in the revised manuscript are highlighted in red for ease of reference.

We hope these additions help address the concerns raised and make the contributions and limitations easier to assess. Please let us know if any questions or concerns remain; we would be happy to continue the discussion and provide further clarification. If our responses and revision has resolved your concerns, we would be grateful if you could consider updating your assessment and rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.