- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the degradation of OOD generalization during SFT, which prior studies associate with changes in dominant singular subspaces. To address the computational cost of identifying loss-sensitive subspaces, the authors propose RPSFT, motivated by the concentration of empirical-Fisher gradient energy in leading pretrained coordinates. RPSFT softly anchors the top-k singular-vector product blocks in the pretrained SVD basis, reducing dominant-subspace rotation while preserving task adaptation. Experiments across different model families and sizes demonstrate that RPSFT improves the balance between in-domain adaptation and OOD generalization and provides stronger initializations for downstream reinforcement learning.
Critical strengths
1.Preserving OOD generalization during supervised fine-tuning is highly relevant, particularly when balancing downstream adaptation with the retention of pretrained capabilities.
2.Experiments with DAPO and GRPO suggest that RPSFT provides stronger SFT initializations, leading to improved downstream generalization.
3.The authors evaluate different model families and tasks, supported by comparisons with existing methods, ablation studies, and geometric analyses. These experiments provide substantial empirical support for the effectiveness of RPSFT.
Critical weaknesses
1.The proposed regularization does not directly guarantee rotation preservation. RPSFT penalizes weight changes within the leading singular-vector product block, but cross-subspace updates remain unconstrained and may still rotate the leading singular subspaces.
2.The reliance on full-parameter fine-tuning limits practical efficiency. RPSFT retains all trainable parameters while introducing additional SVD-based regularization. Although the reported computational overhead is moderate, its training and memory requirements may limit applicability to larger models.
3.The discussion of related orthogonal fine-tuning methods is incomplete. Methods such as OFT [1], BOFT [2], and PSOFT [3] have explored structured orthogonal transformations for fine-tuning. Although their objectives differ from RPSFT, discussing these approaches would help clarify the relationship between orthogonal adaptation and the proposed spectral protection mechanism.
[1] Qiu et al. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023.
[2] Liu et al. Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR 2024.
[3] Wu et al. Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation. ICLR 2026.
Questions for authors
1.Why does higher gradient energy justify protecting leading singular subspaces? Could the authors provide further evidence linking gradient sensitivity to OOD degradation, beyond the observed energy concentration?
2.How does RPSFT ensure rotation preservation when cross-subspace updates remain unconstrained? Could the authors further clarify the theoretical relationship between the proposed regularization and the observed reduction in subspace rotation
3.How should the protected rank (k) be selected? Given variations in matrix dimensions, model architectures, and downstream tasks, could the authors provide practical guidance on rank selection?
4.Can RPSFT be effectively integrated with PEFT methods? Although Appendix A.2 provides a LoRA formulation, experimental comparisons would help establish whether the proposed regularization retains its benefits under more resource-efficient fine-tuning settings.
7 comments
Response (2/3)
Q1: Why protect coordinates with high gradient energy?
Gradient energy motivates the choice of coordinates; the restoration and training controls provide evidence of their relevance to retention.
Sensitivity. Figure 2 shows that a leading block occupying roughly 5% of the SVD core captures roughly 20% of empirical gradient energy in the measured layer-1 query projections.
Restoration. Table 17's hybrid-weight intervention replaces selected SFT directions with pretrained directions. At rank 768, leading-direction restoration raises Qwen2.5-7B instruction-following accuracy from 41.59% to 44.23%, versus 28.85% for rank-matched random restoration averaged over five random-index seeds. The choice of pretrained directions therefore matters for the recovered OOD performance.
Training controls. Table 6 compares different protection mechanisms under matched protection rank and training settings. Mean OOD accuracy over three training seeds is:
| Adaptation task | Hard leading-block protection | Random-block regularization | RPSFT |
|---|---|---|---|
| CSQA | 34.10 | 34.99 | 38.61 |
| ProofWriter | 29.43 | 33.09 | 39.43 |
The OOD mean averages IFEval-loose, MMLU-Pro, and SuperGPQA. RPSFT improves retention over both controls on both tasks. These results support using a soft anchor on the leading pretrained block.
Q3: Selecting the protected rank
Our default is for an weight matrix, with . The cap 768 corresponds to roughly one-fifth of the common 4096- and 3584-dimensional bases, and the rank sweep supports it as a practical default.
Tables 21–22 show that the preferred rank depends on the model and objective. Math-adaptation results are:
| Model | Protected rank | ID Avg@k | OOD Avg@1 |
|---|---|---|---|
| Llama-3.1-8B | 512 | 26.50 | 43.66 |
| Llama-3.1-8B | 768 | 26.01 | 45.19 |
| Llama-3.1-8B | 2048 | 23.97 | 42.99 |
| Qwen2.5-7B | 512 | 38.88 | 52.97 |
| Qwen2.5-7B | 768 | 39.20 | 53.15 |
| Qwen2.5-7B | 2048 | 38.39 | 54.57 |
The sweep makes the trade-off concrete: Qwen's rank 2048 improves OOD over 768 but lowers ID, while Llama declines on both. We therefore use 768 as a practical default and clip it to each matrix's dimensions. For a new task or architecture, and can be adjusted according to the desired validation ID/OOD balance. We will add this guidance to the method description.
Response (3/3)
W2/Q4: LoRA integration and practical cost
RPSFT can regularize a LoRA update without making the frozen base weights trainable, as formulated in Appendix A.2. We now provide results with Qwen2.5-3B, LoRA rank 32, scaling parameter , protected rank 768, and greedy decoding. All results below use step 500. OOD is the equal-weight mean of IFEval loose, MMLU-Pro, and SuperGPQA; all scores are percentages.
| Task | Method | ID | IFEval loose | MMLU-Pro | SuperGPQA | OOD mean |
|---|---|---|---|---|---|---|
| CSQA | LoRA | 73.63 | 17.01 | 38.57 | 13.58 | 23.05 |
| CSQA | + RPSFT, | 81.16 | 28.65 | 37.14 | 15.31 | 27.03 |
| CSQA | + RPSFT, | 81.00 | 28.47 | 41.43 | 15.80 | 28.57 |
| CSQA | + RPSFT, | 81.82 | 27.36 | 38.57 | 16.79 | 27.57 |
| CSQA | + RPSFT, | 81.33 | 24.95 | 38.57 | 15.06 | 26.20 |
| ProofWriter | LoRA | 78.99 | 23.84 | 42.86 | 18.52 | 28.41 |
| ProofWriter | + RPSFT, | 79.64 | 24.95 | 44.29 | 18.52 | 29.25 |
| ProofWriter | + RPSFT, | 87.94 | 28.65 | 37.14 | 17.78 | 27.86 |
| ProofWriter | + RPSFT, | 85.60 | 24.95 | 40.00 | 19.51 | 28.15 |
| ProofWriter | + RPSFT, | 83.20 | 26.62 | 41.43 | 20.49 | 29.51 |
At the shared strength , RPSFT improves ID/OOD accuracy by 7.70/3.15 points on CSQA and 4.21/1.10 points on ProofWriter. All four CSQA strengths improve both means; on ProofWriter, and 0.5 trade a small OOD decrease for higher ID accuracy. This provides initial evidence that RPSFT retains its benefits when integrated with LoRA.
We also profiled training memory on eight H200 GPUs with Qwen2.5-3B/CSQA, BF16, ZeRO-2 with CPU optimizer offload, batch 16, and a common measurement window of optimizer steps 6–50:
| Method | Trainable parameters | Peak training memory, GiB/GPU |
|---|---|---|
| Full SFT | 3,085.94M | 10.17 |
| Full RPSFT, analytical implementation | 3,085.94M | 13.26 |
| LoRA | 59.87M | 6.50 |
| LoRA + RPSFT, analytical implementation | 59.87M | 9.70 |
LoRA + RPSFT retains LoRA's 59.87M trainable parameters and uses 3.56 GiB/GPU less training memory than full RPSFT. The persistent auxiliary storage is approximately 25% of the storage of the regularized weight matrices in this configuration. The regularizer adds about 3.2 GiB/GPU to vanilla LoRA and about 3.1 GiB/GPU to full SFT in measured peak training memory.
Separately, Table 7 reports 156.3 versus 165.7 GPU-hours for the original full-tuning SFT and RPSFT experiments, a 6.0% increase. The LoRA results show that the regularizer can also be used with a much smaller trainable-parameter budget.
W3: Relationship to OFT, BOFT, and PSOFT
We added the following distinction to the related-work discussion in the revision:
| Method | Adaptation and geometric constraint |
|---|---|
| OFT | Applies orthogonal transformations to pretrained weights, preserving pairwise neuron angles and hyperspherical energy. |
| BOFT | Parameterizes orthogonal transformations with butterfly factors to improve parameter efficiency. |
| PSOFT | Adapts within a pretrained principal subspace; establishes geometry preservation under an orthogonality condition and uses learned scaling vectors to relax it. |
| RPSFT | Penalizes accumulated displacement in a fixed leading product block. Cross blocks remain free; the penalty can accompany full tuning or LoRA. |
RPSFT's distinguishing feature is its soft penalty on accumulated displacement in fixed pretrained product coordinates. This retains cross-block adaptation and applies to both full tuning and LoRA, while the other methods organize adaptation through structured transformation parameterizations.
Response (1/3)
Thank you for your constructive review; we have expanded the related-work discussion in the revised manuscript and provide additional rotation measurements, LoRA experiments, and memory comparisons below.
W1/Q2: Unconstrained cross updates and the theoretical scope
We agree that cross-subspace updates can rotate the leading singular subspaces. RPSFT uses soft anchoring to reduce rotation empirically; it does not enforce exact subspace invariance. Appendix C.3 explicitly accounts for the cross-block effects.
Its direct effect is soft anchoring. For fixed pretrained bases, let Ac=Uk⊤(W−W0)Vk and A=UkAcVk⊤. The penalty gradient is 2λA. Writing the task loss as f(W), Appendix C.4 gives, at a stationary point of the full-tuning objective with λ>0,
=\frac{\|U_k^\top\nabla f(W^\star)V_k\|_F}{2\lambda}.$$ This formalizes the restoring effect on the leading block. For a prefix $r<k$, $A$ also contains couplings across the rank-$r$ boundary, providing a mechanism through which anchoring can reduce rotation. New checkpoint measurements support this selective anchoring. We decomposed Qwen2.5-3B math updates using the pretrained rank-768 basis, across 15 matrices: the query, output, gate, up, and down projections in layers 0, 17, and 35. Here, $B,C$ are the two cross blocks. | Update component | SFT joint norm | RPSFT joint norm | SFT energy share | RPSFT energy share | |---|---:|---:|---:|---:| | $A$ | 1.104 | 0.094 | 4.74% | 0.07% | | $B+C$ | 3.448 | 2.493 | 46.21% | 46.52% | Joint norm pools the Frobenius norms across the matrices; energy share is the fraction of total squared update norm. RPSFT reduces the leading-block norm from 1.104 to 0.094, while the cross blocks retain a similar share of update energy. This confirms the selectivity of the anchor. We tested this using Qwen2.5-3B math checkpoints with $k=768$, across 15 query/output/gate/up/down matrices in layers 0, 17, and 35. For each matrix, we rescaled the full SFT and RPSFT updates to the smaller Frobenius norm, then measured angles relative to pretrained weights. Values average each matrix's mean angle; $U,V$ denote left and right subspaces. | Measured prefix $r$ | SFT $U$ | RPSFT $U$ | SFT $V$ | RPSFT $V$ | |---:|---:|---:|---:|---:| | 8 | 0.382° | 0.260° | 0.330° | 0.179° | | 32 | 0.477° | 0.278° | 0.430° | 0.205° | | 64 | 0.536° | 0.293° | 0.495° | 0.224° | | 128 | 0.559° | 0.318° | 0.519° | 0.253° | | 256 | 0.556° | 0.331° | 0.518° | 0.273° | | 512 | 0.536° | 0.358° | 0.496° | 0.306° | RPSFT has smaller mean left and right angles at every reported prefix after controlling total update magnitude. We also measured isolated updates at $r=512$ using the same matrices: | Reconstruction | Norm control | SFT $U$ | RPSFT $U$ | SFT $V$ | RPSFT $V$ | |---|---|---:|---:|---:|---:| | $W_0+A$ | Each method's native component magnitude | 0.467° | 0.051° | 0.466° | 0.051° | | $W_0+B$ | Matched component norm | 0.117° | 0.113° | 0.249° | 0.243° | | $W_0+C$ | Matched component norm | 0.310° | 0.301° | 0.122° | 0.117° | | $W_0+B+C$ | Matched component norm | 0.340° | 0.333° | 0.291° | 0.281° | The learned $A$ component produces substantially less rotation under RPSFT, while cross components have similar responses at equal norms. These results connect the penalty's selective anchoring to the observed reduction in rotation, while confirming that cross-block adaptation remains available.