- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper decomposes the forgetting risk of fine-tuning into the product of the update magnitude and the directional sensitivity, and proposes a fine-tuning formulation that restricts the column space of the update to the dominant left singular subspace of the pre-trained weight in order to control the direction without pre-training data. As implementations, it presents RSP-LoRA, which projects the B matrix of LoRA onto that subspace at every step, and VeFA, which attaches a per-input-dimension scaling vector to the truncated pre-trained weight.
Critical strengths
(S1) The paper is well written and easy to read.
(S2) The motivation of mitigating the forgetting that occurs in LoRA fine-tuning through a subspace constraint is clear.
Critical weaknesses
(W1) The performance improvement is minor. In Table 1, the proposed methods other than VeFA are very similar to the baselines in both performance and forgetting, while VeFA has good performance but worse forgetting. In Table 2, all of the proposed methods perform worse than LoRA. This looks like simply training “less” in order to prevent forgetting, and it raises doubts about the performance of the model.
(W2) Based on prior work showing that intruder dimensions create new directions, the authors allow learning only along the principal component directions of the pre-trained weight. However, MiLoRA, which the authors cite, points out that the important pre-trained information lies in the principal components, and to reduce forgetting it does the opposite: it initializes the adapter with the minor components and keeps the principal components intact. The authors’ method instead induces changes in the principal component region. This comes across as contradictory.
(W3) In general, the principal components obtained by decomposing W and the Fisher curvature computed from the task are different. There is no evidence that the principal components and the low-curvature Fisher directions are the same or similar, and the paper shows this only through experimental performance without actually measuring it. For analysis, this needs to be measured using an open dataset on an actual pre-trained LLM, or shown theoretically.
Questions for authors
(Q1) Does the same effect hold in continual learning? Orthogonal Subspace Learning for Language Model Continual Learning mitigates forgetting by learning in a subspace orthogonal to the existing (adapter) weights, and it needs to be properly compared with the proposed model, together with MiLoRA and others mentioned in (W1). Please explain what difference justifies the proposed approach, and compare in the setting of Table 2 of that paper (CL evaluation on T5-Large).
(Q2) The authors describe RSP-LoRA-EWC and VeFA-EWC as follows: “In our implementation, EWC importance is estimated using only the fine-tuning data and does not require access to the pre-training distribution.” EWC is a regularization term that keeps the parameters from moving away from the task on which the Fisher was measured, so measuring it on the fine-tuning data is contradictory. The original EWC paper measures the Fisher on the pre-training task (or the previous task in continual learning). The authors should explain this.
3 comments
W2: MiLoRA initializes adapters with minor components to preserve the principal components and reduce forgetting, whereas the proposed method allows updates within the dominant pre-trained subspace, inducing changes in the principal component region. These opposite strategies appear contradictory.
A: MiLoRA and our approach are not contradictory. As demonstrated in Section 4.5 (Figures 3–5), restricting directional flexibility itself can reduce forgetting, whether using dominant, bottom, or random subspaces. We select the dominant pre-trained subspace because it maximizes pre-trained weight-energy coverage under a fixed dimensional budget, achieving a more favorable stability–plasticity trade-off, as empirically demonstrated in Figures 3–5 of the revised manuscript.
Methodological difference. MiLoRA uses minor singular components for initialization but does not continuously enforce regularization during training. In contrast, RSP-LoRA maintains the subspace constraint throughout training, and our formulation further allows complementary magnitude regularization.
Empirical comparisons. We have included MiLoRA in our comparisons on GLUE and Qwen2.5-7B.
GLUE (Table 2 of the revised manuscript):
| Method | ID Avg ↑ | Retention Avg ↑ |
|---|---|---|
| LoRA | 84.2 | -0.469 |
| MiLoRA | 84.7 | -0.213 |
| RSP-LoRA-FTP | 83.3 | -0.127 |
| RSP-LoRA-EWC | 82.9 | -0.050 |
Qwen2.5-7B (Table 3 of the revised manuscript):
| Method | Overall ↑ |
|---|---|
| LoRA | 61.64 |
| MiLoRA | 61.42 |
| RSP-LoRA | 62.15 |
| RSP-LoRA-FTP | 62.59 |
These results show that continuous directional control, combined with magnitude regularization, achieves stronger forgetting mitigation than MiLoRA on GLUE, while RSP-LoRA-FTP also achieves better overall performance on Qwen2.5-7B.
W3: The principal components obtained from weight SVD and the low-curvature Fisher directions are generally different. There is no evidence that the principal components and low-curvature Fisher directions are the same or similar. The paper only demonstrates experimental performance without directly measuring their alignment. This relationship should be measured using an open dataset on an actual pre-trained LLM or established theoretically.
A: Dominant weight-SVD directions and low-curvature Fisher directions are fundamentally distinct concepts. Our method neither assumes nor requires their alignment. Since pre-training data are unavailable during fine-tuning, identifying low-curvature Fisher directions is generally infeasible. As clarified in Section 3.1 and our response to W2, our strategy is to restrict directional flexibility while maximizing pre-trained weight-energy coverage under a fixed dimensional budget, rather than relying on Fisher eigenspace alignment. Figures 3–5 demonstrate that dominant, bottom, and random subspace restrictions can all reduce forgetting. Furthermore, Table 18 of the revised manuscript shows that our methods achieve lower estimated directional sensitivity , providing additional empirical support for this strategy.
Q1: Does the same forgetting-mitigation effect hold in continual learning? How does the proposed approach differ from O-LoRA and MiLoRA, and can it be compared with these methods under the T5-large continual learning setting of Table 2 in O-LoRA?
A: We take a cautious view of extending our findings to continual learning. As discussed in our response to W2 of Reviewer 4mFJ, forgetting in sequential continual learning and forgetting during individual downstream fine-tuning are distinct research problems. The former primarily concerns interference among sequentially learned tasks, whereas the latter focuses on preserving pre-trained model structure and capabilities during adaptation.
Extending our method to continual learning would require additional mechanisms to manage task-to-task interference. We therefore do not claim its effectiveness in continual learning and leave such extensions and evaluations for future work.
For methodological differences from O-LoRA, please refer to our response to W2 of Reviewer 4mFJ. For MiLoRA, please refer to our response to W2, including empirical comparisons on GLUE and Qwen2.5-7B.
Q2: Why is EWC importance estimated using only the fine-tuning data rather than the pre-training distribution? EWC is a regularization term that keeps parameters from moving away from the task on which the Fisher was measured. The original EWC estimates Fisher on the pre-training task (or the previous task in continual learning), making the use of fine-tuning data appear contradictory. The authors should explain this choice.
A: We thank the reviewer for highlighting this distinction. In our implementation, the Fisher is estimated using fine-tuning data and used to weight a quadratic penalty on deviations from the pre-trained parameters, providing data-dependent magnitude regularization without requiring access to the original pre-training corpus. Importantly, these Fisher weights are not intended to represent parameter importance under the pre-training distribution, distinguishing our approach from standard EWC.
We acknowledge that the EWC label is imprecise and will rename this component fine-tuning-data Fisher-weighted regularization (FWR) in the next manuscript revision. For consistency with the current tables and reviewer comments, we retain the existing "EWC" labels in this rebuttal.
We greatly thank the reviewer for the comments.
W1 (Weakness item #1): The performance improvement is minor and may simply result from training "less" to prevent forgetting.
W1.1: In Table 1, the proposed methods other than VeFA are very similar to the baselines in both performance and forgetting, while VeFA has good performance but worse forgetting.
A: We respectfully disagree with this assessment. The CLIP results should be evaluated through two groups of comparisons: (1) lightweight and post-hoc baselines in few-shot learning, and (2) LoRA variants with matched trainable parameter counts. The first group has minimal adaptation flexibility (very few parameters or strong alignment with the pretrained model), which is an inherent advantage in few-shot learning. The second group isolates the effect of joint directional and magnitude control at the same parameter budget.
Lightweight and post-hoc baselines. With only 16 examples per class, trainable parameter count can significantly influence downstream performance and forgetting. VeFA-EWC achieves the best retention (-0.59) with higher in-domain accuracy (88.2%) than the lightweight VeRA (-0.83, 87.4%) and post-hoc LoRA-WISE (-0.91, 88.1%).
Matched LoRA parameter counts. Under the same rank-one LoRA adapter budget, RSP-LoRA-EWC improves both retention (-2.41 vs. -3.39) and in-domain accuracy (88.4% vs. 87.8%) compared with LoRA-EWC. Similarly, RSP-LoRA-FTP substantially improves retention over LoRA-FTP (-2.86 vs. -5.19), with comparable in-domain performance.
These results demonstrate the effectiveness of jointly controlling update magnitude and directional flexibility, with directional control providing additional forgetting-mitigation benefits beyond magnitude regularization alone.
W1.2: In Table 2, all proposed methods perform worse than LoRA, raising doubts about the downstream performance of the model.
A: We respectfully disagree with this assessment. Our results demonstrate a favorable stability–plasticity trade-off, achieving substantial forgetting reduction with only minor decreases in downstream performance.
Compared with LoRA, RSP-LoRA-FTP reduces in-domain performance by only 1.1% (84.2 to 83.3), while reducing forgetting by 72.9% (0.469 to 0.127). Similarly, RSP-LoRA-EWC incurs only a 1.5% relative decrease in in-domain performance (84.2 to 82.9), while reducing forgetting by 89.3% (0.469 to 0.050).
These substantial forgetting reductions with minimal downstream performance degradation are particularly valuable for applications that prioritize reliability and the preservation of pre-trained capabilities.
W1.3: The forgetting improvements may simply result from training "less" rather than the proposed subspace constraint.
A: We respectfully disagree with this explanation. Our additional performance-matched experiments demonstrate that reducing the LoRA training budget does not reproduce the forgetting-mitigation benefits of our subspace constraint.
Performance-matched comparison (Table 20 of the revised manuscript). We compare RSP-LoRA with early-stopped LoRA on four GLUE tasks, matching checkpoints based solely on downstream performance.
Across seven matched pairs, the mean absolute downstream-performance difference is only 0.06 points, while RSP-LoRA reduces average forgetting by 36.5% (0.4040 to 0.2564).
These results provide direct empirical evidence that our forgetting improvements are not merely due to training less, but reflect the additional benefit of restricting directional flexibility.