acceptodds
Under review as a conference paper at ICLR 2027

A New Forgetting-Risk Constrained Optimization for Model Fine-Tuning

Abstract

Fine-tuning a large pre-trained model is often necessary for strong downstream performance, but can degrade generalization and cause catastrophic forgetting of pre-trained knowledge. Existing forgetting-mitigation methods commonly regularize parameter changes or control selected spectral structures, such as suppressing intruder dimensions or protecting top singular components. Such component-wise control may provide limited spectral coverage, as unprotected directions can become prominent during adaptation and contribute to forgetting. Motivated by the observation that unconstrained adaptation can develop prominent task-specific spectral directions associated with forgetting, we regularize directional flexibility by restricting updates to an admissible subspace. We formulate forgetting-aware fine-tuning with joint adaptation-subspace and update-magnitude constraints, using the dominant column subspace of the pre-trained weights as a data-free, model-informed choice. We develop two realizations: Representation-Space Projected LoRA (RSP-LoRA), which projects LoRA updates onto the selected subspace, and Vector-based Feature Adaptation (VeFA), which satisfies the constraint through feature-side parameterization. Across image classification, natural language understanding, and Qwen2.5-7B mathematical reasoning, our methods substantially mitigate forgetting while maintaining competitive downstream performance. Ablations show that subspace restriction itself reduces forgetting, while the dominant pre-trained subspace generally provides a more favorable stability–plasticity trade-off than dimension-matched random and bottom subspaces.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Fine-Tuning, Forgetting Risk, Constrained Optimization, Representation-Space Preservation, Magnitude Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ynCWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper decomposes the forgetting risk of fine-tuning into the product of the update magnitude and the directional sensitivity, and proposes a fine-tuning formulation that restricts the column space of the update to the dominant left singular subspace of the pre-trained weight in order to control the direction without pre-training data. As implementations, it presents RSP-LoRA, which projects the B matrix of LoRA onto that subspace at every step, and VeFA, which attaches a per-input-dimension scaling vector to the truncated pre-trained weight.

Critical strengths

(S1) The paper is well written and easy to read.

(S2) The motivation of mitigating the forgetting that occurs in LoRA fine-tuning through a subspace constraint is clear.

Critical weaknesses

(W1) The performance improvement is minor. In Table 1, the proposed methods other than VeFA are very similar to the baselines in both performance and forgetting, while VeFA has good performance but worse forgetting. In Table 2, all of the proposed methods perform worse than LoRA. This looks like simply training “less” in order to prevent forgetting, and it raises doubts about the performance of the model.

(W2) Based on prior work showing that intruder dimensions create new directions, the authors allow learning only along the principal component directions of the pre-trained weight. However, MiLoRA, which the authors cite, points out that the important pre-trained information lies in the principal components, and to reduce forgetting it does the opposite: it initializes the adapter with the minor components and keeps the principal components intact. The authors’ method instead induces changes in the principal component region. This comes across as contradictory.

(W3) In general, the principal components obtained by decomposing W and the Fisher curvature computed from the task are different. There is no evidence that the principal components and the low-curvature Fisher directions are the same or similar, and the paper shows this only through experimental performance without actually measuring it. For analysis, this needs to be measured using an open dataset on an actual pre-trained LLM, or shown theoretically.

Questions for authors

(Q1) Does the same effect hold in continual learning? Orthogonal Subspace Learning for Language Model Continual Learning mitigates forgetting by learning in a subspace orthogonal to the existing (adapter) weights, and it needs to be properly compared with the proposed model, together with MiLoRA and others mentioned in (W1). Please explain what difference justifies the proposed approach, and compare in the setting of Table 2 of that paper (CL evaluation on T5-Large).

(Q2) The authors describe RSP-LoRA-EWC and VeFA-EWC as follows: “In our implementation, EWC importance is estimated using only the fine-tuning data and does not require access to the pre-training distribution.” EWC is a regularization term that keeps the parameters from moving away from the task on which the Fisher was measured, so measuring it on the fine-tuning data is contradictory. The original EWC paper measures the Fisher on the pre-training task (or the previous task in continual learning). The authors should explain this.

3 comments

We greatly thank the reviewer for the comments.


W1 (Weakness item #1): The performance improvement is minor and may simply result from training "less" to prevent forgetting.

W1.1: In Table 1, the proposed methods other than VeFA are very similar to the baselines in both performance and forgetting, while VeFA has good performance but worse forgetting.

A: We respectfully disagree with this assessment. The CLIP results should be evaluated through two groups of comparisons: (1) lightweight and post-hoc baselines in few-shot learning, and (2) LoRA variants with matched trainable parameter counts. The first group has minimal adaptation flexibility (very few parameters or strong alignment with the pretrained model), which is an inherent advantage in few-shot learning. The second group isolates the effect of joint directional and magnitude control at the same parameter budget.

Lightweight and post-hoc baselines. With only 16 examples per class, trainable parameter count can significantly influence downstream performance and forgetting. VeFA-EWC achieves the best retention (-0.59) with higher in-domain accuracy (88.2%) than the lightweight VeRA (-0.83, 87.4%) and post-hoc LoRA-WISE (-0.91, 88.1%).

Matched LoRA parameter counts. Under the same rank-one LoRA adapter budget, RSP-LoRA-EWC improves both retention (-2.41 vs. -3.39) and in-domain accuracy (88.4% vs. 87.8%) compared with LoRA-EWC. Similarly, RSP-LoRA-FTP substantially improves retention over LoRA-FTP (-2.86 vs. -5.19), with comparable in-domain performance.

These results demonstrate the effectiveness of jointly controlling update magnitude and directional flexibility, with directional control providing additional forgetting-mitigation benefits beyond magnitude regularization alone.


W1.2: In Table 2, all proposed methods perform worse than LoRA, raising doubts about the downstream performance of the model.

A: We respectfully disagree with this assessment. Our results demonstrate a favorable stability–plasticity trade-off, achieving substantial forgetting reduction with only minor decreases in downstream performance.

Compared with LoRA, RSP-LoRA-FTP reduces in-domain performance by only 1.1% (84.2 to 83.3), while reducing forgetting by 72.9% (0.469 to 0.127). Similarly, RSP-LoRA-EWC incurs only a 1.5% relative decrease in in-domain performance (84.2 to 82.9), while reducing forgetting by 89.3% (0.469 to 0.050).

These substantial forgetting reductions with minimal downstream performance degradation are particularly valuable for applications that prioritize reliability and the preservation of pre-trained capabilities.


W1.3: The forgetting improvements may simply result from training "less" rather than the proposed subspace constraint.

A: We respectfully disagree with this explanation. Our additional performance-matched experiments demonstrate that reducing the LoRA training budget does not reproduce the forgetting-mitigation benefits of our subspace constraint.

Performance-matched comparison (Table 20 of the revised manuscript). We compare RSP-LoRA with early-stopped LoRA on four GLUE tasks, matching checkpoints based solely on downstream performance.

TaskRSP α\alphaLoRA BudgetLoRA ID ↑RSP-LoRA ID ↑LoRA Forgetting ↓RSP-LoRA Forgetting ↓
STS-B0.7040%90.52190.4770.8559000.223891
STS-B0.9960%90.70890.8050.7920680.616571
MRPC0.7050%91.39891.3270.4130070.276716
CoLA0.9060%59.15659.0520.1487600.109348
CoLA0.9960%59.15659.0550.1487600.152173
RTE0.7010%74.72974.7290.1519950.163034
RTE0.9550%77.25677.2560.3175790.252963
Mean--77.56177.5290.4040100.256385

Across seven matched pairs, the mean absolute downstream-performance difference is only 0.06 points, while RSP-LoRA reduces average forgetting by 36.5% (0.4040 to 0.2564).

These results provide direct empirical evidence that our forgetting improvements are not merely due to training less, but reflect the additional benefit of restricting directional flexibility.

W2: MiLoRA initializes adapters with minor components to preserve the principal components and reduce forgetting, whereas the proposed method allows updates within the dominant pre-trained subspace, inducing changes in the principal component region. These opposite strategies appear contradictory.

A: MiLoRA and our approach are not contradictory. As demonstrated in Section 4.5 (Figures 3–5), restricting directional flexibility itself can reduce forgetting, whether using dominant, bottom, or random subspaces. We select the dominant pre-trained subspace because it maximizes pre-trained weight-energy coverage under a fixed dimensional budget, achieving a more favorable stability–plasticity trade-off, as empirically demonstrated in Figures 3–5 of the revised manuscript.

Methodological difference. MiLoRA uses minor singular components for initialization but does not continuously enforce regularization during training. In contrast, RSP-LoRA maintains the subspace constraint throughout training, and our formulation further allows complementary magnitude regularization.

Empirical comparisons. We have included MiLoRA in our comparisons on GLUE and Qwen2.5-7B.

GLUE (Table 2 of the revised manuscript):

MethodID Avg ↑Retention Avg ↑
LoRA84.2-0.469
MiLoRA84.7-0.213
RSP-LoRA-FTP83.3-0.127
RSP-LoRA-EWC82.9-0.050

Qwen2.5-7B (Table 3 of the revised manuscript):

MethodOverall ↑
LoRA61.64
MiLoRA61.42
RSP-LoRA62.15
RSP-LoRA-FTP62.59

These results show that continuous directional control, combined with magnitude regularization, achieves stronger forgetting mitigation than MiLoRA on GLUE, while RSP-LoRA-FTP also achieves better overall performance on Qwen2.5-7B.


W3: The principal components obtained from weight SVD and the low-curvature Fisher directions are generally different. There is no evidence that the principal components and low-curvature Fisher directions are the same or similar. The paper only demonstrates experimental performance without directly measuring their alignment. This relationship should be measured using an open dataset on an actual pre-trained LLM or established theoretically.

A: Dominant weight-SVD directions and low-curvature Fisher directions are fundamentally distinct concepts. Our method neither assumes nor requires their alignment. Since pre-training data are unavailable during fine-tuning, identifying low-curvature Fisher directions is generally infeasible. As clarified in Section 3.1 and our response to W2, our strategy is to restrict directional flexibility while maximizing pre-trained weight-energy coverage under a fixed dimensional budget, rather than relying on Fisher eigenspace alignment. Figures 3–5 demonstrate that dominant, bottom, and random subspace restrictions can all reduce forgetting. Furthermore, Table 18 of the revised manuscript shows that our methods achieve lower estimated directional sensitivity Dˉ\bar D, providing additional empirical support for this strategy.


Q1: Does the same forgetting-mitigation effect hold in continual learning? How does the proposed approach differ from O-LoRA and MiLoRA, and can it be compared with these methods under the T5-large continual learning setting of Table 2 in O-LoRA?

A: We take a cautious view of extending our findings to continual learning. As discussed in our response to W2 of Reviewer 4mFJ, forgetting in sequential continual learning and forgetting during individual downstream fine-tuning are distinct research problems. The former primarily concerns interference among sequentially learned tasks, whereas the latter focuses on preserving pre-trained model structure and capabilities during adaptation.

Extending our method to continual learning would require additional mechanisms to manage task-to-task interference. We therefore do not claim its effectiveness in continual learning and leave such extensions and evaluations for future work.

For methodological differences from O-LoRA, please refer to our response to W2 of Reviewer 4mFJ. For MiLoRA, please refer to our response to W2, including empirical comparisons on GLUE and Qwen2.5-7B.

Q2: Why is EWC importance estimated using only the fine-tuning data rather than the pre-training distribution? EWC is a regularization term that keeps parameters from moving away from the task on which the Fisher was measured. The original EWC estimates Fisher on the pre-training task (or the previous task in continual learning), making the use of fine-tuning data appear contradictory. The authors should explain this choice.

A: We thank the reviewer for highlighting this distinction. In our implementation, the Fisher is estimated using fine-tuning data and used to weight a quadratic penalty on deviations from the pre-trained parameters, providing data-dependent magnitude regularization without requiring access to the original pre-training corpus. Importantly, these Fisher weights are not intended to represent parameter importance under the pre-training distribution, distinguishing our approach from standard EWC.

We acknowledge that the EWC label is imprecise and will rename this component fine-tuning-data Fisher-weighted regularization (FWR) in the next manuscript revision. For consistency with the current tables and reviewer comments, we retain the existing "EWC" labels in this rebuttal.

Reviewer_4mFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of catastrophic forgetting during parameter-efficient fine-tuning (PEFT) of large pre-trained models. The authors formulate a second-order approximation of the forgetting risk, decomposing it into total update magnitude (MM) and directional sensitivity (Dˉ\bar{D}). Because pre-training data are typically inaccessible to estimate the Fisher Information Matrix (FptF_{\text{pt}}), the authors propose regularizing the directional flexibility of adaptation by restricting updates to a data-free, model-informed admissible subspace—specifically, the dominant column subspace of pre-trained weights (S0(α)=Col(Urα)S_0^{(\alpha)} = \text{Col}(U_{r_\alpha})) determined by retained spectral energy α\alpha.

To realize this formulation, the authors propose two mechanisms: (1) Representation-Space Projected LoRA (RSP-LoRA), which projects low-rank updates onto the retained pre-trained singular subspace via orthogonal projection; and (2) Vector-based Feature Adaptation (VeFA), which parameterizes adaptation through direct feature scaling to inherently satisfy the subspace constraint. The authors evaluate their methods across three distinct domains: 16-shot image classification with CLIP ViT-B/16, natural language understanding on GLUE with RoBERTa-Base, and mathematical reasoning with Qwen2.5-7B on MetaMathQA. Extensive ablation studies evaluate the stability–plasticity trade-off against dimension-matched random and bottom subspaces, as well as rank-one magnitude-matched controls.

Critical strengths
  1. Practical, Data-Free Formulation with Negligible Serving Overhead: By using the dominant left-singular subspace of the frozen pre-trained weights rather than requiring access to historical pre-training datasets or expensive online curvature estimations, the proposed approach is practical and computationally lightweight. The Singular Value Decomposition (SVD) is performed only once prior to fine-tuning (taking under 90 seconds across all 140 target matrices in Qwen2.5-7B), adds minimal training overhead (7–8%), and can be merged back into the base backbone weights upon deployment, resulting in zero additional latency or memory footprint during inference.
  2. Thoughtful Controlled Experiments and Multi-Domain Evaluation: The empirical study spans diverse modalities and architectures, including multimodal vision (CLIP), language understanding (RoBERTa), and instruction fine-tuning on a 7B LLM (Qwen2.5). More importantly, the authors conduct meaningful control experiments—such as benchmarking against dimension-matched random and bottom subspaces (Figure 2) and providing rank-one magnitude-matched comparisons (Figure 3)—which demonstrate that confining updates to the dominant pre-trained subspace yields a more favorable stability–plasticity trade-off than arbitrary dimensional restriction alone.
Critical weaknesses
  1. Theoretical Oversimplification of Deep Transformer Dynamics and Missing Multi-Layer Coupling: The central theoretical justification in Section 3.1 relies on a second-order Taylor expansion (Equation 2) with a strict layer-wise block-diagonal Fisher assumption: RF(Wft)≈12∑ℓ=1Lvec(Δ(ℓ))⊤Fpt(ℓ)vec(Δ(ℓ))R_F(W_{\text{ft}}) \approx \frac{1}{2}\sum_{\ell=1}^L \text{vec}(\Delta^{(\ell)})^\top F_{\text{pt}}^{(\ell)} \text{vec}(\Delta^{(\ell)}). In deep multi-layer Transformers (e.g., 28–32 layers in Qwen2.5-7B), parameter updates in lower layers propagate nonlinearly through alternating Self-Attention, LayerNorm, and MLP blocks, fundamentally altering the input representations for subsequent layers. Consequently, cross-layer Hessian terms (∂2L∂W(i)∂W(j)\frac{\partial^2 \mathcal{L}}{\partial W^{(i)} \partial W^{(j)}}) are non-negligible, and layer-isolated second-order approximations do not straightforwardly translate to total task loss changes.

Furthermore, the paper explicitly notes on page 4 that the adopted subspace S0(α)S_0^{(\alpha)} is neither assumed to approximate the low-curvature eigenspace of FptF_{\text{pt}} nor guaranteed to be optimal for minimizing forgetting. Treating each layer's weight SVD in isolation without accounting for cross-layer representation drift leaves an evident disconnect between the formal risk factorization and the empirical projection heuristic.

  1. Omission of Key Orthogonal LoRA Literature in the Main Text and Baseline Comparisons: The idea of projecting LoRA updates onto orthogonal or interference-minimizing subspaces is not without precedent. Highly relevant and influential works on subspace-constrained adaptation, such as O-LoRA (Wang et al., 2023) and InfLoRA (Liang & Li, 2024), are completely absent from the main text and are only briefly mentioned in a single passing sentence in Appendix A (page 14). While the authors delineate their setup as single-task pre-trained model retention rather than sequential continual learning, the underlying mathematical mechanism—constraining parameter-efficient adapters to null or dominant spectral subspaces to prevent representation interference—shares identical conceptual foundations.

Omitting these prominent orthogonal LoRA methods from both the main-text literature review and the empirical comparisons weakens the contextualization of the paper's novelty. Crucially, the only related subspace baseline included, OPLoRA (Xiong & Xie, 2026), appears untuned in Table 3 (scoring 61.66±0.2861.66 \pm 0.28 vs. standard LoRA's 61.64±0.1661.64 \pm 0.16), leaving it unclear how the proposed method compares against well-established orthogonal PEFT techniques.

Questions for authors
  1. Clarification on Cross-Layer Coupling in Deep Transformers: Could the authors elaborate on how cross-layer interactions affect the validity of the layer-wise block-diagonal Fisher approximation in deep Transformers? Specifically, does restricting ΔW(ℓ)\Delta W^{(\ell)} independently per layer sufficiently prevent representation collapse when the cumulative shift across 28+ cascaded layers is considered? An empirical check on layer-wise feature divergence would be very helpful.
  2. Comparison with Orthogonal LoRA Frameworks (O-LoRA, InfLoRA): Given that O-LoRA and InfLoRA also project low-rank updates to minimize interference with frozen representations, how does RSP-LoRA fundamentally differ from adapting their projection principles to the single-task pre-training retention setting? Could the authors provide an empirical comparison with InfLoRA or an adapted orthogonal LoRA baseline on the CLIP or RoBERTa benchmarks?
  3. Clarification on Table 17 Arithmetic: In Appendix Table 17, the estimated risk is defined as R^F=12MDˉ\hat{R}_F = \frac{1}{2} M \bar{D}. For LoRA-EWC, the reported numbers (M=1554,Dˉ=1.20×10−5M = 1554, \bar{D} = 1.20 \times 10^{-5}) closely match the reported R^F=0.0093\hat{R}_F = 0.0093. However, for the LoRA row, M=1723M = 1723 and Dˉ=3.22×10−5\bar{D} = 3.22 \times 10^{-5} would yield 12×1723×(3.22×10−5)≈0.0277\frac{1}{2} \times 1723 \times (3.22 \times 10^{-5}) \approx 0.0277, whereas the table reports R^F=0.0167\hat{R}_F = 0.0167. Could the authors kindly clarify whether this discrepancy is due to task-level averaging, differing normalizations, or a typographical error?
  4. Missing Variants in GLUE Benchmark (Table 2): Could the authors provide the results for standalone VeFA and RSP-LoRA-FTP on the GLUE benchmark? Clarifying whether standalone VeFA maintains competitive in-domain accuracy on natural language tasks would help readers better understand the general applicability of feature-side adaptation beyond few-shot vision settings.
  5. Configuration Clarification for Figure 2 and Table 4: Table 4 lists a LoRA rank of r=1r=1 as the base configuration for CLIP image experiments, whereas the caption of Figure 2 mentions an ablation conducted with LoRA rank r=8r=8. Could the authors clarify whether the ablation in Figure 2 was conducted on a specific subset of datasets or under a distinct rank setting?
3 comments

We greatly thank the reviewer for the comments.


W1.1: The central theoretical justification in Section 3.1 relies on a second-order Taylor expansion (Equation 2) with a strict layer-wise block-diagonal Fisher assumption. However, in deep multi-layer Transformers, parameter updates in lower layers propagate nonlinearly through alternating Self-Attention, LayerNorm, and MLP blocks, fundamentally altering the input representations for subsequent layers. Consequently, cross-layer Hessian terms ∂2L∂W(i)∂W(j)\frac{\partial^2\mathcal{L}}{\partial W^{(i)}\partial W^{(j)}} are non-negligible, and layer-isolated second-order approximations do not straightforwardly translate to total task loss changes.

A: We thank the reviewer for pointing out the limitation of our original forgetting-risk estimation. To address this limitation, we have relaxed the block-diagonal assumption and extended our analysis to the full-network Fisher/GGN matrix, explicitly accounting for cross-layer interactions. Our magnitude–directional sensitivity factorization remains valid under this more general formulation.

Furthermore, we have added a scatter plot of estimated forgetting risk versus measured forgetting (Figure 2 of the revised manuscript). Our proposed methods continue to exhibit both low estimated forgetting risk and low measured forgetting, providing further empirical support for our approach after accounting for cross-layer interactions.

For details of the generalization to the full Fisher/GGN matrix, please refer to the first two paragraphs of Section 3.1 of the revised manuscript.


W1.2: The adopted subspace S0(α)S_0^{(\alpha)} is neither assumed to approximate the low-curvature eigenspace of FptF_{\mathrm{pt}} nor guaranteed to be optimal for minimizing forgetting. Treating each layer's weight SVD in isolation without accounting for cross-layer representation drift leaves an evident disconnect between the formal risk factorization and the empirical projection heuristic.

A: Our key strategy is to restrict layer-wise directional flexibility, together with complementary update-magnitude control, to mitigate forgetting. Although the subspace constraints are imposed at each layer, our empirical results demonstrate their effectiveness in reducing cumulative representation drift and full-network forgetting.

Representation preservation. We evaluate cumulative representation drift across all 28 Qwen2.5-7B decoder blocks using fixed general-domain text. The final-block results are summarized below:

MethodCKA divergence ↓Normalized squared feature drift ↓
LoRA0.0295810.017591
LoRA-XS0.1006470.067312
MiLoRA0.0831750.073104
RSP-LoRA-FTP0.0219840.011649

RSP-LoRA-FTP reduces final-block CKA divergence and normalized squared feature drift by 25.7% and 33.8%, respectively, compared with LoRA. Figure 9 of the revised manuscript presents the layer-wise results, with the detailed analysis provided in Appendix E.5.

Empirical evidence against the claimed disconnect. Importantly, our forgetting evaluation captures the cumulative effects of all adapted layers. Table 18 of the revised manuscript shows that RSP-LoRA reduces both full-network estimated risk (0.0395 to 0.0107) and measured forgetting (0.469 to 0.266) compared with LoRA. Similar reductions are observed when adding our subspace constraint to FTP (0.0121 to 0.0056 and 0.264 to 0.127, respectively). Figure 2 of the revised manuscript further shows that our proposed methods exhibit both low estimated risk and low measured forgetting. Together with the representation-drift analysis, these results provide direct empirical evidence connecting our layer-wise subspace constraints to reduced full-network forgetting, even in the presence of cross-layer interactions.

W2: Highly relevant works on subspace-constrained adaptation, such as O-LoRA and InfLoRA, are absent from the main text and only briefly mentioned in Appendix A. It is unclear how the proposed method compares against well-established orthogonal PEFT techniques.

A: Although subspace-constrained adaptation methods such as O-LoRA, InfLoRA, and our approach share similar conceptual foundations, their objectives differ substantially across problem settings. In our single-task fine-tuning setting, the stability–plasticity trade-off is crucial. We therefore restrict directional flexibility to mitigate forgetting and select the dominant pre-trained subspace to maximize weight-energy coverage under a fixed dimensional budget, helping preserve adaptation flexibility. We discuss the key differences below:

Adapting InfLoRA to our setting. Extending InfLoRA's historical-task protection mechanism to preserve pre-trained knowledge would require pre-training gradient-subspace information, which is often unavailable during fine-tuning. We therefore evaluate InfLoRA using its original first-task configuration, where the input-side subspace is fixed based on downstream activations and only AA is trained in ΔW=AB\Delta W=AB, without historical-task protection. On GLUE (Table 2), RSP-LoRA-FTP achieves substantially better retention (-0.127 vs. -1.404) while maintaining competitive in-domain performance (83.3% vs. 85.0%).

Adapting O-LoRA's orthogonality principle to protect dominant pre-trained singular directions would restrict updates to the complementary bottom subspace, corresponding to our Bottom-SP LoRA under a matched dimensional budget. However, our ablations (Figures 3–5) demonstrate that forgetting can be mitigated by restricting directional flexibility itself, without requiring orthogonality to dominant pre-trained directions. Dominant, bottom, and even random subspace projections generally reduce forgetting during fine-tuning. Moreover, by restricting updates to the dominant pre-trained representation subspace, RSP-LoRA achieves a substantially more favorable stability–plasticity trade-off than Bottom-SP LoRA as shown in Figures 3–5 of the revised manuscript.

For space considerations and to maintain the logical flow of the main text, we provide a detailed discussion of both methods in Appendix A of the revised manuscript.


Q1: How do cross-layer interactions affect the validity of the layer-wise block-diagonal Fisher approximation in deep Transformers? Does restricting ΔW(ℓ)\Delta W^{(\ell)} independently per layer sufficiently prevent representation collapse when cumulative shifts across 28+ cascaded layers are considered? An empirical check on layer-wise feature divergence would be helpful.

A: Please refer to our response to W1.1 and W1.2 for the detailed analysis.


Q2: How does RSP-LoRA fundamentally differ from adapting the projection principles of O-LoRA and InfLoRA to the single-task pre-training retention setting? Could the authors provide an empirical comparison with InfLoRA or an adapted orthogonal LoRA baseline on CLIP or RoBERTa?

A: Please refer to our response to W2 for the detailed methodological differences and our additional empirical comparison with InfLoRA on GLUE (Table 2 of the revised manuscript).


Q3: Clarification on Table 17 Arithmetic. Could the authors kindly clarify whether this discrepancy is due to task-level averaging, differing normalizations, or a typographical error?

A: The discrepancy arises from task-level averaging, rather than differing normalizations or a typographical error. The formula R^F=12MDˉ\hat R_F=\frac12 M\bar D holds for each individual task, while the reported quantities are averaged separately across six GLUE tasks. We have clarified this in Table 18 of the revised manuscript.

Q4: Missing Variants in GLUE Benchmark (Table 2): Could the authors provide the results for standalone VeFA and RSP-LoRA-FTP on the GLUE benchmark? Clarifying whether standalone VeFA maintains competitive in-domain accuracy on natural language tasks would help readers better understand the general applicability of feature-side adaptation beyond few-shot vision settings.

A: We thank the reviewer for this suggestion. We have added both standalone VeFA and RSP-LoRA-FTP to Table 2 of the revised manuscript.

MethodID Avg ↑Retention Avg ↑
LoRA84.2-0.469
LoRA-FTP81.6-0.264
VeFA82.4-0.236
VeFA-EWC81.7-0.074
RSP-LoRA-FTP83.3-0.127

Standalone VeFA achieves competitive in-domain performance (82.4%) while substantially reducing forgetting compared with LoRA (-0.236 vs. -0.469), demonstrating its applicability beyond few-shot vision settings. Meanwhile, RSP-LoRA-FTP improves both in-domain performance and retention over LoRA-FTP.

Applicability across data regimes. Our two realizations offer complementary stability–plasticity trade-offs for different data regimes and applications. In particular, VeFA combined with magnitude control is especially effective in few-shot settings, as demonstrated by VeFA-EWC on 16-shot CLIP (88.2% ID accuracy and -0.59 retention, Table 1). Its lightweight and more restrictive feature-wise adaptation also offers promising potential for data-scarce and safety-critical applications where preserving pre-trained capabilities is particularly important.


Q5: Table 4 lists a LoRA rank of r=1r=1 as the base configuration for CLIP image experiments, whereas Figure 2 reports an ablation with r=8r=8. Was the Figure 2 ablation conducted on a specific subset of datasets or under a distinct rank setting?

A: The ablation in Figure 2 of the original manuscript (Figures 3–5 of the revised manuscript) was conducted using RoBERTa-Base on four GLUE tasks (STS-B, MRPC, CoLA, and RTE) with LoRA rank r=8r=8, rather than CLIP. Table 4 specifies the CLIP experimental configuration with r=1r=1, so these are distinct experimental settings. We have clarified the GLUE setting in the figure captions and Section 4.5 of the revised manuscript to avoid confusion.

Reviewer_ZWwqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies forgetting induced by fine-tuning. A second-order expansion of the pre-training loss gives a forgetting risk that factorizes into a total squared update magnitude MM and a magnitude-weighted directional sensitivity Dˉ\bar D, so that R^F=12MDˉ\hat R_F=\frac12 M\bar D. Because the pre-training Fisher is treated as unavailable, the authors do not estimate curvature. They instead restrict the fine-tuning update to the dominant column subspace S0S_0 of the pre-trained weights, whose dimension rαr_\alpha is set by a retained spectral-energy ratio α\alpha, and write this restriction together with an update-magnitude bound as a constrained problem. Two realizations are proposed. RSP-LoRA projects the LoRA factor BB onto S0S_0 after each optimizer step, and VeFA adds a diagonal scaling branch built on the truncated weights Wpt(α)W_{pt}^{(\alpha)} with zero initialization. Experiments cover CLIP ViT-B/16 on seven image-classification datasets, RoBERTa-Base on six GLUE tasks, and Qwen2.5-7B on mathematical reasoning.

Critical strengths
  1. The rank-one controlled comparison is the strongest part of the paper. Section 4.5(2) and Figure 3 match both parameter count and update magnitude, which pre-empts the usual objection that the gains come from a smaller update.
  2. Evaluation spans three model families with measurement matched to available data. Forgetting is assessed by held-out datasets for CLIP, by MLM pseudo-loss on sampled pre-training text for RoBERTa, and by held-out reasoning benchmarks for Qwen.
  3. Reporting is candid. Method-specific hyperparameters, SVD cost, memory overhead, and the fact that EWC importance is estimated on fine-tuning data are all disclosed.
Critical weaknesses
  1. The risk model is never validated, and the table supporting it contradicts the stated formula. The paper defines R^F=12MDˉ\hat R_F=\frac12M\bar D, but Table 17 violates it: for LoRA, 12⋅1723⋅3.22×10−5=0.0277\frac12\cdot1723\cdot3.22\times10^{-5}=0.0277 against a reported 0.01670.0167, and the table even inverts its own ordering, with VeFA-EWC carrying the higher estimated risk but the lower measured forgetting. The Fisher estimator is never specified, and R^F\hat R_F is never plotted against the measured RFR_F although both are available. The only support is a Spearman ρ=0.554\rho=0.554 across methods, which leaves most of the variance unexplained and cannot carry the central claim that forgetting risk decomposes into two separately controllable factors.
  2. The subspace constraint and the intruder definition operate at different scales. Definition 2.1 identifies an intruder relative to the top-kk singular directions, but kk is never given, no intruder statistics are reported, and the constraint instead spans rα≈383r_\alpha\approx383 of 768 dimensions at α=0.9\alpha=0.9. What the constraint provably gives is narrower: since ΔW=PΔW\Delta W=P\Delta W, the tail is frozen, so nothing orthogonal to S0S_0 exceeds σrα+1\sigma_{r_\alpha+1}, while the spectrum inside a half-dimensional S0S_0 can still be reordered freely. The per-layer intruder counts that would test the stated mechanism are absent, and RSP-LoRA applies the constraint only for the first 5% of training, so its final weights no longer satisfy it.
  3. The figures do not follow the paper's typography, and none of them describes the method. The figures are set in fonts that do not match the paper. The text is small enough to be hard to read on screen. More importantly, no figure describes the method itself, so the reader must reconstruct from prose how S0S_0 is formed, where the projection is applied, and how VeFA differs from RSP-LoRA. One overview figure covering the spectral split and the two realizations would resolve most of this.
  4. The theoretical and methodological advances over related spectral adaptation methods remain unclear. The Fisher-based second-order approximation, magnitude–directional sensitivity factorization, and optimal weight-energy coverage of the dominant spectral subspace largely rely on standard results and do not establish that the selected subspace reduces forgetting risk. Moreover, the absence of comparisons with directly related spectral adaptation methods, such as LoRA-XS and SVFT, makes the independent value of the proposed constraint difficult to assess. The paper should clarify its substantive differences from these methods and evaluate adaptation performance and retention under matched budgets.
Questions for authors
  1. What is the exact definition of MM in Table 17, why does only one of its four rows satisfy R^F=12MDˉ\hat R_F=\frac12M\bar D, and how is the Fisher estimated? Since pre-training text is sampled for GLUE, can you plot R^F\hat R_F against measured RFR_F with a regression and an R2R^2?
  2. Can you report per-layer intruder counts and the cosine similarity of the strongest intruder for LoRA, RSP-LoRA, and VeFA? At α=0.9\alpha=0.9, can reordering inside S0S_0 still produce intruders relative to a small top-kk?
  3. Does the RSP-LoRA checkpoint in Figure 3 satisfy Col(ΔW)⊆S0\mathrm{Col}(\Delta W)\subseteq S_0, given that the constraint is removed after 5% of training?
  4. Which model and dataset does Figure 2 use, how is its LoRA value of 79.657% reconciled with 84.2% in Table 16, and why is α=0.9\alpha=0.9 used when Appendix F favors 0.3?
3 comments

We greatly thank the reviewer for the comments.


W1.1: Table 17 contradicts the stated formula. The paper defines R^F=12MDˉ\hat{R}_F = \frac{1}{2}M\bar{D}, but Table 17 violates it: for LoRA, $ \frac{1}{2}\cdot1723\cdot3.22\times10^{-5}=0.0277, $ against a reported 0.01670.0167.

A: We apologize for the confusion caused by the notation. Table 17 in the original submission (renumbered as Table 18 in the revised manuscript) reports each quantity averaged separately over six GLUE tasks to provide an overall summary of each method's behavior. We have updated the column notation to ⟨M⟩t\langle M\rangle_t, ⟨Dˉ⟩t\langle\bar D\rangle_t, and ⟨R^F⟩t\langle\widehat R_{\mathrm F}\rangle_t, where ⟨X⟩t=16∑t=16Xt\langle X\rangle_t=\frac{1}{6}\sum_{t=1}^{6}X_t.


W1.2: Table 17 inverts its own ordering. VeFA-EWC carries the higher estimated risk but the lower measured forgetting. The Fisher estimator is never specified. R^F\hat{R}_F is not plotted against the measured RFR_F although both are available for GLUE. The Spearman correlation across methods does not explain most of the variance.

A: This is a good observation. Estimated forgetting risk does not necessarily follow the same ranking as measured forgetting across methods. Some methods have low estimated risk but relatively high measured forgetting, and vice versa. An effective forgetting-mitigation methods should achieve both low estimated risk and low measured forgetting.

We have added a scatter plot of R^F\widehat R_F versus measured RFR_F on GLUE (Figure 2 of the revised manuscript). Notably, our proposed methods including VeFA-EWC are concentrated in the lower-left region, indicating both low estimated forgetting risk and low observed forgetting.

Fisher estimation. Following Frantar et al. (2021), we use the widely adopted gradient outer-product estimator to estimate the full-network empirical Fisher. The detailed estimation procedure is provided in Appendix E.3 of the revised manuscript.


W2.1: The subspace constraint and the intruder definition operate at different scales. Definition 2.1 identifies an intruder relative to the top-kk singular directions, but kk is never given. The constraint instead spans rα≈383r_\alpha \approx 383 of 768768 dimensions at α=0.9\alpha=0.9. What the constraint provably gives is narrower: since ΔW=PΔW\Delta W=P\Delta W, the tail is frozen, so nothing orthogonal to S0S_0 exceeds σrα+1\sigma_{r_\alpha+1}, while the spectrum inside a half-dimensional S0S_0 can still be reordered freely. This does not guarantee a reduction in intruder count, and the per-layer intruder counts that would test the stated mechanism are absent.

A: Although our work is inspired by intruder dimensions, we aim to address two limitations of existing spectral-control methods, including the method of reducing intruder count. First, intruder suppression operates post hoc and does not directly constrain updates during fine-tuning. Second, methods that protect selected leading spectral components may fail as the importance of individual singular directions changes during adaptation. To address these limitations, we restrict updates to a much broader pre-trained spectral subspace and jointly control update magnitude during fine-tuning to mitigate forgetting.

Clarification of different scales. In our paper, we use k=10k=10 for intruder detection, whereas rαr_\alpha denotes the substantially larger, matrix-dependent adaptation-subspace dimension (e.g., r0.9=322r_{0.9}=322 for a representative RoBERTa-Base matrix on GLUE, and an average r0.9≈1,704r_{0.9}\approx1{,}704 across 140 adapted matrices for Qwen2.5-7B). They serve different purposes, and our constraint does not guarantee fewer intruders. The detection threshold ϵ=0.5\epsilon=0.5 and post-hoc scaling factor of 0.70.7 are specified in Tables 5 and 11. Additional intruder-count analyses are provided in Appendix E.6 (Table 20 and Figure 9).


W2.2: In Appendix E.4, RSP-LoRA applies the constraint only for the first 5% of training, so its final weights no longer satisfy it.

A: The subspace constraint is maintained throughout training by construction, since RSP-LoRA parameterizes the update as ΔW=UrαUrα⊤BA\Delta W=U_{r_\alpha}U_{r_\alpha}^\top BA. Therefore, the final checkpoint satisfies Col⁡(ΔW)⊆S0\operatorname{Col}(\Delta W)\subseteq\mathcal S_0.

The 5% schedule applies only to magnitude matching. For training stability, magnitude matching is disabled during the first 5% of training, gradually introduced over the next 5%, and fully enforced thereafter. The subspace constraint remains active throughout. We have clarified this distinction in the first paragraph of Appendix E.4 of the revised manuscript.

W3.1: The figures do not follow the paper's typography, and none of them describes the method.

A: We thank the reviewer for this helpful suggestion. We have added a new method overview (Figure 1) illustrating our forgetting-aware constrained optimization formulation, the SVD-based construction of the dominant pre-trained subspace, and its two realizations: projection-based RSP-LoRA and feature-side VeFA.


W3.2: The figures are set in fonts that do not match the paper. The text is small enough to be hard to read on screen.

A: We thank the reviewer for pointing out these presentation issues. In the revised manuscript, we have increased the font sizes to improve readability.


W4.1: The Fisher-based second-order approximation, magnitude–directional sensitivity factorization, and optimal weight-energy coverage of the dominant spectral subspace do not establish that the selected subspace reduces forgetting risk.

A: We respectfully disagree with this comment. Through extensive controlled experiments, we demonstrate that restricting directional flexibility can reduce forgetting risk, while complementary magnitude control provides further benefits.

Empirical evidence. Figures 3–5 show that restricting directional flexibility reduces forgetting even with random and bottom subspaces, while the dominant pre-trained subspace achieves a more favorable stability–plasticity trade-off. Figure 6 further demonstrates forgetting reduction under matched parameter count and update magnitude.

Direct risk validation. Table 18 of the revised manuscript shows that RSP-LoRA reduces both estimated forgetting risk (0.0395 to 0.0107) and observed forgetting (0.469 to 0.266) compared with LoRA. Similarly, RSP-LoRA-FTP improves both quantities over LoRA-FTP (0.0121 to 0.0056 and 0.264 to 0.127, respectively). Figure 2 further shows that our proposed methods are concentrated in the lower-left region, exhibiting both low estimated forgetting risk and low observed forgetting.

Together with comprehensive baseline comparisons across three benchmarks, these results provide direct empirical evidence that constraining updates to the dominant pre-trained subspace can reduce forgetting risk, and that jointly controlling directional flexibility and update magnitude achieves stronger forgetting mitigation than existing baselines during fine-tuning.


W4.2: The absence of comparisons with directly related spectral adaptation methods, such as LoRA-XS and SVFT, makes the independent value of the proposed constraint difficult to assess. The paper should clarify its substantive differences from these methods and evaluate adaptation performance and retention under matched budgets.

A: Existing spectral adaptation methods, such as LoRA-XS and SVFT, are fundamentally different from our approach. They are primarily PEFT methods designed to improve parameter efficiency and downstream performance, without explicitly targeting forgetting mitigation or providing guarantees for pre-trained knowledge retention. In contrast, our approach explicitly controls directional flexibility and update magnitude to mitigate forgetting during fine-tuning.

Empirical comparisons. We have added LoRA-XS and SVFT as baselines. On 16-shot CLIP (Table 1), VeFA-EWC achieves substantially better retention (-0.59) than LoRA-XS (-5.49) and SVFT (-2.55), while maintaining competitive in-domain accuracy (88.2% vs. 86.4% and 89.3%). On GLUE (Table 2), RSP-LoRA-EWC, VeFA-EWC, and RSP-LoRA-FTP achieve the three best retention results (-0.050, -0.074, and -0.127), outperforming LoRA-XS (-0.304) and SVFT (-0.451). On Qwen2.5-7B (Table 3), RSP-LoRA-FTP achieves a higher overall score than LoRA-XS (62.59 vs. 62.20).

Q1: What is the exact definition of MM in Table 17, why does only one of its four rows satisfy R^F=12MDˉ\hat{R}_F=\frac{1}{2}M\bar{D}, and how is the Fisher estimated? Since pre-training text is sampled for GLUE, can you plot R^F\hat{R}_F against measured RFR_F with a regression and an R2R^2?

A: Please refer to our response to W1.1 and W1.2.


Q2: Can you report per-layer intruder counts and the cosine similarity of the strongest intruder for LoRA, RSP-LoRA, and VeFA? At α=0.9\alpha=0.9, can reordering inside S0S_0 still produce intruders relative to a small top-kk?

A: Please refer to our responses to W2.


Q3: Does the RSP-LoRA checkpoint in Figure 3 satisfy Col⁡(ΔW)⊆S0\operatorname{Col}(\Delta W)\subseteq S_0, given that the constraint is removed after 5% of training?

A: Yes. The subspace constraint is maintained throughout training, including at the final checkpoint. Please refer to our response to W2.3 and the clarified Appendix E.4 of the revised manuscript.


Q4: Which model and dataset does Figure 2 use, how is its LoRA value of 79.657% reconciled with 84.2% in Table 16, and why is α=0.9\alpha=0.9 used when Appendix F favors 0.3?

A: Experimental setting. Figures 3–5 of the revised manuscript use RoBERTa-Base on four GLUE tasks (STS-B, MRPC, CoLA, and RTE) to reduce computational cost, as the ablation requires testing multiple α\alpha values. Thus, the LoRA reference of 79.657% is averaged over four tasks, whereas 84.2% in the original Table 16 is averaged over all six GLUE tasks.

Choice of α\alpha. We use α=0.9\alpha=0.9 as a fixed default across our main experiments, without task-specific hyperparameter tuning of α\alpha. Even with this fixed choice, our methods achieve leading forgetting mitigation and competitive downstream performance compared with comprehensive baselines. Adaptively selecting α\alpha based on task characteristics is a promising direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.