- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Subspace Tuning (SST) to mitigate catastrophic forgetting during LLM fine-tuning. SST computes optimizer updates from full gradients, then projects them onto the non-principal singular subspaces of weight matrices using a two-sided projection. It optionally refreshes the protected subspaces to balance adaptation and retention.
Critical strengths
Potential benefits for capability retention. In Table 5, no-refresh SST achieves similar average in-domain accuracy to SFT, with better retention of prior capabilities. These results suggest that SST may offer a useful adaptation–retention trade-off in the evaluated settings.
Critical weaknesses
-
Limited methodological novelty. OPLoRA already uses the same two-sided projection, and Adam-NSCL already projects Adam-generated updates. Compared with OPLoRA, SST optimizes full weights instead of LoRA factors and optionally refreshes the subspace. The paper does not clearly establish why these changes constitute a substantial methodological advance.
-
The benefit of basis refresh is not well explained. The paper motivates refresh through subspace rotation, yet exact two-sided updates preserve the protected singular directions. If these directions remain the leading ones, recomputing the SVD should leave the projector unchanged. The authors should show how the protected subspace changes during training and explain how these changes produce the reported gains.
Questions for authors
- What is SST’s main technical contribution beyond OPLoRA’s two-sided projection? In the matched-protection comparison with OPLoRA, what adapter rank was used, and could limited adapter capacity explain the remaining performance gap?
4 comments
Q1 What is SST’s main technical contribution beyond OPLoRA’s two-sided projection? In the matched-protection comparison with OPLoRA, what adapter rank was used, and could limited adapter capacity explain the remaining performance gap?
R1 SST computes optimizer statistics from full weight gradients, then constrains the actual dense update, with optional basis refresh. OPLoRA instead optimizes factor-coordinate gradients and states; it is not equivalent to projecting gradients and updates in direct weight space. We provide precise theoretical and empirical evidence supporting our claims in Section 4.1 and Appendix B.9; where OPLoRA performs substantially worse at matched subspace settings (r = 0.99d) as well as a larger protection setting (r=128).
| Method | Protected rank k | Adapter r / α | Science | FinQA | Pathology | Medicine | Mean |
|---|---|---|---|---|---|---|---|
| OPLoRA, official setting | 128 | 32 / 32 | 78.5 / 79.1 | 72.8 / 61.7 | 77.4 / 51.6 | 70.0 / 60.0 | 74.7 / 63.1 |
| OPLoRA, matched protection | ⌊0.99d⌋ | 32 / 32 | 80.8 / 79.2 | 71.5 / 72.3 | 76.2 / 70.7 | 70.8 / 70.0 | 74.8 / 73.1 |
| SST/R100 | ⌊0.99d⌋ | Dense; no adapter | 81.7 / 80.0 | 73.1 / 80.1 | 77.6 / 78.5 | 74.3 / 78.3 | 76.7 / 79.2 |
These results along with the results provided in R1 further improves the comparison of SST with OPLoRA at scale.
We are committed to addressing all of your concerns and have made our best effort to respond to each of them thoroughly. We would be more than happy to provide further clarification or address any remaining concerns. We would greatly appreciate it if you could consider our efforts and kindly reconsider your review rating in light of our responses.
Thank you for the response. However, my main concern remains. SST mainly combines existing ideas from OPLoRA and Adam-NSCL, and the rebuttal does not show that this combination is a substantial methodological contribution. I will therefore keep my score.
Rebuttal to Reviewer wJyA
W1 Limited methodological novelty. OPLoRA already uses the same two-sided projection, and Adam-NSCL already projects Adam-generated updates. Compared with OPLoRA, SST optimizes full weights instead of LoRA factors and optionally refreshes the subspace. The paper does not clearly establish why these changes constitute a substantial methodological advance.
R1 We thank the reviewer for raising the differences between SST and prior art. SST combines weight-derived spectral protection with full-gradient optimizer statistics, projection of the actual dense update, and an optional refresh mechanism, with broad empirical evidence for the learning–retention trade-off.
OPLoRA optimizes low-rank factors; Adam-NSCL protects previous-task activation subspaces. SST applies optimizer-update projection directly to full weights using subspaces derived from the pretrained weights, without requiring previous-task examples. These differences might seem minor but we present very strong empirical evidence and thorough analysis leading to real gains.
SST’s practical benefit is visible across domains: matched OPLoRA averages 74.8 / 73.1, versus SST(R100’s) 76.7 / 79.2, which improves both metrics in every domain as presented in Table 5. SST scales substantially better than prior art (such as OPLoRA) as shown in Figure 15 in the Appendix.
Upon the feedback of the reviewer, we also provide an updated FinQA ablation that supports the SST design: at identical 76.3 ID, SST retains 75.21, versus 54.58 when both gradients and updates are projected.
Table C. FinQA projection ablation.
We have also added scaling experiments on Medicine and Pathology too in the Rebuttal Section C in the Appendix of the draft. We can clearly see that OPLoRA starts to collapse as the data scales. We hope that our in depth comparison with prior art as presented in Appendix B.9 as well as the new ablations clearly demonstrates the contribution of SST.
W2 The benefit of basis refresh is not well explained. The paper motivates refresh through subspace rotation, yet exact two-sided updates preserve the protected singular directions. If these directions remain the leading ones, recomputing the SVD should leave the projector unchanged. The authors should show how the protected subspace changes during training and explain how these changes produce the reported gains.
R2 We agree with the reviewer’s conditional statement: if the protected directions remain leading and the cutoff is separated, exact refresh leaves the projector unchanged.
However, preserving those directions does not preserve their ranking. Singular values in the free subspace can grow past the protected cutoff. Refresh then protects the new leading set and releases previously protected directions for adaptation. This changes the allowed update space, while keeping its dimension fixed, and provides a mechanism for changing the adaptation–retention balance.
We measured these changes through training steps.
For an orthonormal protected basis U, its projector is P=UU⊤. The distance between the old and refreshed projectors is
D = \left\\|P_{\mathrm{new}} - P_{\mathrm{old}}\right\\|_F = \sqrt{\sum_i \sum_j \left[(P_{\mathrm{new}})_{ij} - (P_{\mathrm{old}})_{ij}\right]^2}.
If one unit direction u is replaced by an orthogonal unit direction v, with the other directions unchanged, then Pnew−Pold=vv⊤−uu⊤. Each direction's projector has squared Frobenius norm 1, because ∣uu⊤∣F2=(u⊤u)2=1 for a unit vector, and similarly for v. Their cross-term is zero because u⊤v=0. Therefore,
D2=∣uu⊤∣F2+∣vv⊤∣F2−2(u⊤v)2=1+1−0=2,
D=2≈1.414.
A value near 1.414 represents the same total subspace change as one orthogonal direction replacement. D = 0 means an unchanged span; D > 1.414 indicates significant change in the protected basis
Thus, refresh changes the constraint during training, especially early in this run. By protecting the current leading directions and releasing earlier ones, it allows subsequent adaptation to follow the updated spectrum.