acceptodds
Under review as a conference paper at ICLR 2027

Rotation-Preserving Supervised Fine-Tuning

11 views

Abstract

Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work relates this degradation to changes in dominant singular subspaces of pretrained weight matrices. Identifying dominant loss-sensitive subspaces from full Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose Rotation-Preserving Supervised Fine-Tuning (RPSFT), which anchors the dominant top- product block in the pretrained SVD basis, motivated by the concentration of empirical-Fisher gradient energy in leading pretrained coordinates. This block-level anchor empirically reduces dominant-subspace rotation while preserving task adaptation. Across model families and sizes, RPSFT improves the in-domain/OOD trade-off over standard SFT and strong SFT baselines on math and non-math adaptation, and provides stronger initializations for downstream RL fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised Fine-tuning, Post-training, Large language Models, OOD Forgetting, Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Vx9DOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the degradation of OOD generalization during SFT, which prior studies associate with changes in dominant singular subspaces. To address the computational cost of identifying loss-sensitive subspaces, the authors propose RPSFT, motivated by the concentration of empirical-Fisher gradient energy in leading pretrained coordinates. RPSFT softly anchors the top-k singular-vector product blocks in the pretrained SVD basis, reducing dominant-subspace rotation while preserving task adaptation. Experiments across different model families and sizes demonstrate that RPSFT improves the balance between in-domain adaptation and OOD generalization and provides stronger initializations for downstream reinforcement learning.

Critical strengths

1.Preserving OOD generalization during supervised fine-tuning is highly relevant, particularly when balancing downstream adaptation with the retention of pretrained capabilities.

2.Experiments with DAPO and GRPO suggest that RPSFT provides stronger SFT initializations, leading to improved downstream generalization.

3.The authors evaluate different model families and tasks, supported by comparisons with existing methods, ablation studies, and geometric analyses. These experiments provide substantial empirical support for the effectiveness of RPSFT.

Critical weaknesses

1.The proposed regularization does not directly guarantee rotation preservation. RPSFT penalizes weight changes within the leading singular-vector product block, but cross-subspace updates remain unconstrained and may still rotate the leading singular subspaces.

2.The reliance on full-parameter fine-tuning limits practical efficiency. RPSFT retains all trainable parameters while introducing additional SVD-based regularization. Although the reported computational overhead is moderate, its training and memory requirements may limit applicability to larger models.

3.The discussion of related orthogonal fine-tuning methods is incomplete. Methods such as OFT [1], BOFT [2], and PSOFT [3] have explored structured orthogonal transformations for fine-tuning. Although their objectives differ from RPSFT, discussing these approaches would help clarify the relationship between orthogonal adaptation and the proposed spectral protection mechanism.

[1] Qiu et al. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023.

[2] Liu et al. Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR 2024.

[3] Wu et al. Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation. ICLR 2026.

Questions for authors

1.Why does higher gradient energy justify protecting leading singular subspaces? Could the authors provide further evidence linking gradient sensitivity to OOD degradation, beyond the observed energy concentration?

2.How does RPSFT ensure rotation preservation when cross-subspace updates remain unconstrained? Could the authors further clarify the theoretical relationship between the proposed regularization and the observed reduction in subspace rotation

3.How should the protected rank (k) be selected? Given variations in matrix dimensions, model architectures, and downstream tasks, could the authors provide practical guidance on rank selection?

4.Can RPSFT be effectively integrated with PEFT methods? Although Appendix A.2 provides a LoRA formulation, experimental comparisons would help establish whether the proposed regularization retains its benefits under more resource-efficient fine-tuning settings.

7 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for your constructive review; we have expanded the related-work discussion in the revised manuscript and provide additional rotation measurements, LoRA experiments, and memory comparisons below.

W1/Q2: Unconstrained cross updates and the theoretical scope

We agree that cross-subspace updates can rotate the leading singular subspaces. RPSFT uses soft anchoring to reduce rotation empirically; it does not enforce exact subspace invariance. Appendix C.3 explicitly accounts for the cross-block effects.

Its direct effect is soft anchoring. For fixed pretrained bases, let Ac=Uk⊤(W−W0)VkA_c=U_k^\top(W-W_0)V_k and A=UkAcVk⊤A=U_kA_cV_k^\top. The penalty gradient is 2λA2\lambda A. Writing the task loss as f(W)f(W), Appendix C.4 gives, at a stationary point of the full-tuning objective with λ>0\lambda>0,

=\frac{\|U_k^\top\nabla f(W^\star)V_k\|_F}{2\lambda}.$$ This formalizes the restoring effect on the leading block. For a prefix $r<k$, $A$ also contains couplings across the rank-$r$ boundary, providing a mechanism through which anchoring can reduce rotation. New checkpoint measurements support this selective anchoring. We decomposed Qwen2.5-3B math updates using the pretrained rank-768 basis, across 15 matrices: the query, output, gate, up, and down projections in layers 0, 17, and 35. Here, $B,C$ are the two cross blocks. | Update component | SFT joint norm | RPSFT joint norm | SFT energy share | RPSFT energy share | |---|---:|---:|---:|---:| | $A$ | 1.104 | 0.094 | 4.74% | 0.07% | | $B+C$ | 3.448 | 2.493 | 46.21% | 46.52% | Joint norm pools the Frobenius norms across the matrices; energy share is the fraction of total squared update norm. RPSFT reduces the leading-block norm from 1.104 to 0.094, while the cross blocks retain a similar share of update energy. This confirms the selectivity of the anchor. We tested this using Qwen2.5-3B math checkpoints with $k=768$, across 15 query/output/gate/up/down matrices in layers 0, 17, and 35. For each matrix, we rescaled the full SFT and RPSFT updates to the smaller Frobenius norm, then measured angles relative to pretrained weights. Values average each matrix's mean angle; $U,V$ denote left and right subspaces. | Measured prefix $r$ | SFT $U$ | RPSFT $U$ | SFT $V$ | RPSFT $V$ | |---:|---:|---:|---:|---:| | 8 | 0.382° | 0.260° | 0.330° | 0.179° | | 32 | 0.477° | 0.278° | 0.430° | 0.205° | | 64 | 0.536° | 0.293° | 0.495° | 0.224° | | 128 | 0.559° | 0.318° | 0.519° | 0.253° | | 256 | 0.556° | 0.331° | 0.518° | 0.273° | | 512 | 0.536° | 0.358° | 0.496° | 0.306° | RPSFT has smaller mean left and right angles at every reported prefix after controlling total update magnitude. We also measured isolated updates at $r=512$ using the same matrices: | Reconstruction | Norm control | SFT $U$ | RPSFT $U$ | SFT $V$ | RPSFT $V$ | |---|---|---:|---:|---:|---:| | $W_0+A$ | Each method's native component magnitude | 0.467° | 0.051° | 0.466° | 0.051° | | $W_0+B$ | Matched component norm | 0.117° | 0.113° | 0.249° | 0.243° | | $W_0+C$ | Matched component norm | 0.310° | 0.301° | 0.122° | 0.117° | | $W_0+B+C$ | Matched component norm | 0.340° | 0.333° | 0.291° | 0.281° | The learned $A$ component produces substantially less rotation under RPSFT, while cross components have similar responses at equal norms. These results connect the penalty's selective anchoring to the observed reduction in rotation, while confirming that cross-block adaptation remains available.
AuthorsOpenReview ↗

Response (2/3)

Q1: Why protect coordinates with high gradient energy?

Gradient energy motivates the choice of coordinates; the restoration and training controls provide evidence of their relevance to retention.

Sensitivity. Figure 2 shows that a leading block occupying roughly 5% of the SVD core captures roughly 20% of empirical gradient energy in the measured layer-1 query projections.

Restoration. Table 17's hybrid-weight intervention replaces selected SFT directions with pretrained directions. At rank 768, leading-direction restoration raises Qwen2.5-7B instruction-following accuracy from 41.59% to 44.23%, versus 28.85% for rank-matched random restoration averaged over five random-index seeds. The choice of pretrained directions therefore matters for the recovered OOD performance.

Training controls. Table 6 compares different protection mechanisms under matched protection rank and training settings. Mean OOD accuracy over three training seeds is:

Adaptation taskHard leading-block protectionRandom-block regularizationRPSFT
CSQA34.1034.9938.61
ProofWriter29.4333.0939.43

The OOD mean averages IFEval-loose, MMLU-Pro, and SuperGPQA. RPSFT improves retention over both controls on both tasks. These results support using a soft anchor on the leading pretrained block.

Q3: Selecting the protected rank

Our default is k=min⁡(768,m,n)k=\min(768,m,n) for an m×nm\times n weight matrix, with λ=1\lambda=1. The cap 768 corresponds to roughly one-fifth of the common 4096- and 3584-dimensional bases, and the rank sweep supports it as a practical default.

Tables 21–22 show that the preferred rank depends on the model and objective. Math-adaptation results are:

ModelProtected rankID Avg@kOOD Avg@1
Llama-3.1-8B51226.5043.66
Llama-3.1-8B76826.0145.19
Llama-3.1-8B204823.9742.99
Qwen2.5-7B51238.8852.97
Qwen2.5-7B76839.2053.15
Qwen2.5-7B204838.3954.57

The sweep makes the trade-off concrete: Qwen's rank 2048 improves OOD over 768 but lowers ID, while Llama declines on both. We therefore use 768 as a practical default and clip it to each matrix's dimensions. For a new task or architecture, kk and λ\lambda can be adjusted according to the desired validation ID/OOD balance. We will add this guidance to the method description.

AuthorsOpenReview ↗

Response (3/3)

W2/Q4: LoRA integration and practical cost

RPSFT can regularize a LoRA update without making the frozen base weights trainable, as formulated in Appendix A.2. We now provide results with Qwen2.5-3B, LoRA rank 32, scaling parameter α=64\alpha=64, protected rank 768, and greedy decoding. All results below use step 500. OOD is the equal-weight mean of IFEval loose, MMLU-Pro, and SuperGPQA; all scores are percentages.

TaskMethodIDIFEval looseMMLU-ProSuperGPQAOOD mean
CSQALoRA73.6317.0138.5713.5823.05
CSQA+ RPSFT, λ=0.1\lambda=0.181.1628.6537.1415.3127.03
CSQA+ RPSFT, λ=0.3\lambda=0.381.0028.4741.4315.8028.57
CSQA+ RPSFT, λ=0.5\lambda=0.581.8227.3638.5716.7927.57
CSQA+ RPSFT, λ=1\lambda=181.3324.9538.5715.0626.20
ProofWriterLoRA78.9923.8442.8618.5228.41
ProofWriter+ RPSFT, λ=0.1\lambda=0.179.6424.9544.2918.5229.25
ProofWriter+ RPSFT, λ=0.3\lambda=0.387.9428.6537.1417.7827.86
ProofWriter+ RPSFT, λ=0.5\lambda=0.585.6024.9540.0019.5128.15
ProofWriter+ RPSFT, λ=1\lambda=183.2026.6241.4320.4929.51

At the shared strength λ=1\lambda=1, RPSFT improves ID/OOD accuracy by 7.70/3.15 points on CSQA and 4.21/1.10 points on ProofWriter. All four CSQA strengths improve both means; on ProofWriter, λ=0.3\lambda=0.3 and 0.5 trade a small OOD decrease for higher ID accuracy. This provides initial evidence that RPSFT retains its benefits when integrated with LoRA.

We also profiled training memory on eight H200 GPUs with Qwen2.5-3B/CSQA, BF16, ZeRO-2 with CPU optimizer offload, batch 16, and a common measurement window of optimizer steps 6–50:

MethodTrainable parametersPeak training memory, GiB/GPU
Full SFT3,085.94M10.17
Full RPSFT, analytical implementation3,085.94M13.26
LoRA59.87M6.50
LoRA + RPSFT, analytical implementation59.87M9.70

LoRA + RPSFT retains LoRA's 59.87M trainable parameters and uses 3.56 GiB/GPU less training memory than full RPSFT. The persistent auxiliary storage is approximately 25% of the storage of the regularized weight matrices in this configuration. The regularizer adds about 3.2 GiB/GPU to vanilla LoRA and about 3.1 GiB/GPU to full SFT in measured peak training memory.

Separately, Table 7 reports 156.3 versus 165.7 GPU-hours for the original full-tuning SFT and RPSFT experiments, a 6.0% increase. The LoRA results show that the regularizer can also be used with a much smaller trainable-parameter budget.

W3: Relationship to OFT, BOFT, and PSOFT

We added the following distinction to the related-work discussion in the revision:

MethodAdaptation and geometric constraint
OFTApplies orthogonal transformations to pretrained weights, preserving pairwise neuron angles and hyperspherical energy.
BOFTParameterizes orthogonal transformations with butterfly factors to improve parameter efficiency.
PSOFTAdapts within a pretrained principal subspace; establishes geometry preservation under an orthogonality condition and uses learned scaling vectors to relax it.
RPSFTPenalizes accumulated displacement in a fixed leading product block. Cross blocks remain free; the penalty can accompany full tuning or LoRA.

RPSFT's distinguishing feature is its soft penalty on accumulated displacement in fixed pretrained product coordinates. This retains cross-block adaptation and applies to both full tuning and LoRA, while the other methods organize adaptation through structured transformation parameterizations.

Reviewer_ongdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Rotation-Preserving Supervised Fine-Tuning (RPSFT) to mitigate the degradation of out-of-domain capabilities in large language models during supervised fine-tuning. The method defines a protected block using the top-kk left and right singular vectors of the pretrained weight matrices and applies a soft regularization penalty to the accumulated displacement from the pretrained weights within this block, while retaining the flexibility of full-parameter fine-tuning. Experiments on math and non-math adaptation tasks using Llama and Qwen models show that RPSFT improves the trade-off between in-domain performance and out-of-domain capability retention and provides better initializations for subsequent reinforcement learning fine-tuning.

Critical strengths

The paper provides a fairly comprehensive experimental evaluation across several commonly used models, including Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-3B-Instruct, and DeepSeek-7B. It also includes comparisons with a range of baseline methods. The experiements confirm the effectiveness of the proposed method.

Critical weaknesses

1.I find the intuition behind the method somewhat unclear. In particular, the connection between the regularization term and subspace rotation needs further clarification. Why is the proposed regularizer expected to preserve the orientation of the singular subspaces?

  1. Could describing a reduction in rotation as “Rotation-Preserving” be considered an overstatement?

  2. Why should preserving the orientation of the singular subspaces help retain out-of-domain capabilities? Could the authors add an illustrative figure in the introduction to explain this connection?

  3. Minor comment: Figure 1 could be made more visually appealing, with a more compact layout.

Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your helpful suggestions; we have updated both figures in the revised manuscript and clarify the connections between soft anchoring, reduced rotation, and OOD retention below.

W1/W2: Why block anchoring can reduce rotation, and what the name means

The regularizer pulls the leading product block toward its pretrained value. Let Ac=Uk⊤(W−W0)VkA_c=U_k^\top(W-W_0)V_k and A=UkAcVk⊤A=U_kA_cV_k^\top. The penalty λ∥Ac∥F2\lambda\|A_c\|_F^2 has gradient 2λA2\lambda A (Eq. 7; Appendix C.4). The other update components—cross blocks B,CB,C and complementary block DD—remain unpenalized.

For a leading prefix r<kr<k, AA includes couplings between the first rr directions and directions r+1,…,kr+1,\ldots,k. Anchoring AA limits these changes, which can affect subspace rotation, while leaving the cross blocks free for adaptation (Appendix C.3).

New checkpoint measurements support this selective anchoring. We decomposed Qwen2.5-3B math updates using the pretrained rank-768 basis, across 15 matrices: the query, output, gate, up, and down projections in layers 0, 17, and 35. Here, B,CB,C are the two cross blocks.

Update componentSFT joint normRPSFT joint normSFT energy shareRPSFT energy share
AA1.1040.0944.74%0.07%
B+CB+C3.4482.49346.21%46.52%

Joint norm pools the Frobenius norms across the matrices; energy share is the fraction of total squared update norm. RPSFT reduces the leading-block norm from 1.104 to 0.094, while the cross blocks retain a similar share of update energy. This confirms the selectivity of the anchor.

We measured left (UU) and right (VV) principal angles from Qwen2.5-3B math checkpoints, with training rank k=768k=768. To control update magnitude, we rescaled both methods' full updates to the smaller Frobenius norm for each matrix before measuring angles relative to pretrained weights. Results average each matrix's mean angle over 15 query/output/gate/up/down matrices in layers 0, 17, and 35.

Measured prefix rrSFT UURPSFT UUSFT VVRPSFT VV
80.382°0.260°0.330°0.179°
320.477°0.278°0.430°0.205°
640.536°0.293°0.495°0.224°
1280.559°0.318°0.519°0.253°
2560.556°0.331°0.518°0.273°
5120.536°0.358°0.496°0.306°

RPSFT has smaller mean left and right angles at all reported prefixes, even after matching total update magnitude.

To examine the components separately, we also constructed W0+AW_0+A, W0+BW_0+B, W0+CW_0+C, and W0+B+CW_0+B+C and measured angles at r=512r=512:

Isolated componentNorm controlSFT UURPSFT UUSFT VVRPSFT VV
AAEach method's native magnitude0.467°0.051°0.466°0.051°
BBMatched component norm0.117°0.113°0.249°0.243°
CCMatched component norm0.310°0.301°0.122°0.117°
B+CB+CMatched component norm0.340°0.333°0.291°0.281°

The learned AA component produces much less rotation under RPSFT, while cross components have similar responses at equal norms. This matches the intended mechanism: selectively anchor AA while retaining cross-block adaptation.

“Rotation-preserving” refers to this empirical reduction. We will clarify at first use that it does not imply exact subspace invariance.

AuthorsOpenReview ↗

Response (2/2)

W3/W4: OOD motivation and clearer figures

The motivation is to retain pretrained features used beyond the tuning task. Our new schematic illustrates how changing a pretrained direction alters the feature coordinate of the same OOD input. RPSFT softly anchors the leading pretrained mapping while retaining flexibility elsewhere. Three results support this design:

  1. Why use soft leading-block anchoring? Table 6 compares RPSFT with hard leading-block protection and random-block regularization under matched protection rank and training settings. Mean OOD accuracy over three training seeds is:
Adaptation taskHard leading-block protectionRandom-block regularizationRPSFT
CSQA34.1034.9938.61
ProofWriter29.4333.0939.43

The OOD mean averages IFEval-loose, MMLU-Pro, and SuperGPQA. RPSFT improves retention over both controls on both tasks, supporting the combination of leading-coordinate selection and soft protection.

  1. Why are these directions relevant to OOD retention? In Table 17, restoring leading pretrained directions at rank 768 raises Qwen2.5-7B instruction-following accuracy from 41.59% to 44.23%, compared with 28.85% for rank-matched random restoration averaged over five random-index seeds. This intervention provides functional evidence that the choice of pretrained directions matters for OOD performance.
  2. Why protect leading coordinates? Figure 2 shows that a leading block occupying roughly 5% of the SVD core captures about 20% of empirical-Fisher gradient energy in the measured layer-1 query projections. This motivates leading pretrained coordinates as a compact proxy for loss-sensitive directions.

We have prepared an OOD-rationale illustration connecting OOD inputs, Fisher enrichment in leading coordinates, and soft anchoring, together with a compact method overview. We updated both figures in the revision.

Dear Reviewer ongd,

The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.

Best, AC

Reviewer_vPGSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a regularization method designed to mitigate the degradation of out-of-domain (OOD) capabilities during supervised fine-tuning. RPSFT applies an SVD-based regularization term that softly constrains changes in the pretrained top-k singular-vector block while allowing full-parameter adaptation. Experiments across mathematical reasoning and non-math tasks demonstrate improved in-domain/OOD performance trade-offs compared with standard SFT and several retention baselines. Furthermore, RPSFT provides better initialization for downstream reinforcement learning.

Critical strengths

+) Simple and technically reasonable method. The proposed SVD-based regularization is straightforward to implement, preserves full-parameter trainability, and introduces relatively modest computational overhead.

+) Meaningful ablation studies. Comparisons with hard subspace protection, random subspace regularization, and other retention methods support the effectiveness of softly preserving pretrained dominant singular directions.

Critical weaknesses

-) Limited theoretical justification for the proposed regularization.

-) Improvements are sometimes modest. On several benchmarks in Table 1, RPSFT provides relatively small improvements over standard SFT, and it does not consistently outperform all baselines on individual metrics.

Questions for authors

a) How does RPSFT compare with simpler regularization under a matched adaptation–retention trade-off?

b) Do we have any theoretical justification for the proposed regularization?

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for your insightful review; we have clarified the theoretical motivation in the revised manuscript and provide further analysis of the adaptation–retention trade-off below.

W1/Q(b): Theoretical justification

RPSFT directly controls displacement in the leading pretrained product block. For fixed pretrained singular-vector bases, define

Ac=Uk⊤(W−W0)Vk,A=UkAcVk⊤.A_c=U_k^\top(W-W_0)V_k,\qquad A=U_kA_cV_k^\top.

The objective is f(W)+λ∥Ac∥F2f(W)+\lambda\|A_c\|_F^2 (Eq. 7). Its regularizer gradient is 2λA2\lambda A, which pulls this block toward its pretrained value. Appendix C.4 shows that, at a stationary point W⋆W^\star with λ>0\lambda>0,

=\frac{\|U_k^\top\nabla f(W^\star)V_k\|_F}{2\lambda}.$$ This identity formalizes the restoring effect of the regularizer on the leading pretrained block. New checkpoint measurements support this selective anchoring. We decomposed Qwen2.5-3B math updates using the pretrained rank-768 basis, across 15 matrices: the query, output, gate, up, and down projections in layers 0, 17, and 35. Here, $B,C$ are the two cross blocks. | Update component | SFT joint norm | RPSFT joint norm | SFT energy share | RPSFT energy share | |---|---:|---:|---:|---:| | $A$ | 1.104 | 0.094 | 4.74% | 0.07% | | $B+C$ | 3.448 | 2.493 | 46.21% | 46.52% | Joint norm pools the Frobenius norms across the matrices; energy share is the fraction of total squared update norm. RPSFT reduces the leading-block norm from 1.104 to 0.094, while the cross blocks retain a similar share of update energy. This confirms the selectivity of the anchor. For a leading prefix $r<k$, $A$ includes couplings between the first $r$ directions and directions $r+1,\ldots,k$. Anchoring $A$ limits these changes, which can affect subspace rotation, while leaving the cross blocks free for adaptation (Appendix C.3). We measured left ($U$) and right ($V$) principal angles from Qwen2.5-3B math checkpoints, with training rank $k=768$. To control update magnitude, we rescaled both methods' full updates to the smaller Frobenius norm for each matrix before measuring angles relative to pretrained weights. Results average each matrix's mean angle over 15 query/output/gate/up/down matrices in layers 0, 17, and 35. | Measured prefix $r$ | SFT $U$ | RPSFT $U$ | SFT $V$ | RPSFT $V$ | |---:|---:|---:|---:|---:| | 8 | 0.382° | 0.260° | 0.330° | 0.179° | | 32 | 0.477° | 0.278° | 0.430° | 0.205° | | 64 | 0.536° | 0.293° | 0.495° | 0.224° | | 128 | 0.559° | 0.318° | 0.519° | 0.253° | | 256 | 0.556° | 0.331° | 0.518° | 0.273° | | 512 | 0.536° | 0.358° | 0.496° | 0.306° | RPSFT has smaller mean left and right angles at all reported prefixes, even after matching total update magnitude. To examine the components separately, we also constructed $W_0+A$, $W_0+B$, $W_0+C$, and $W_0+B+C$ and measured angles at $r=512$: | Isolated component | Norm control | SFT $U$ | RPSFT $U$ | SFT $V$ | RPSFT $V$ | |---|---|---:|---:|---:|---:| | $A$ | Each method's native magnitude | 0.467° | 0.051° | 0.466° | 0.051° | | $B$ | Matched component norm | 0.117° | 0.113° | 0.249° | 0.243° | | $C$ | Matched component norm | 0.310° | 0.301° | 0.122° | 0.117° | | $B+C$ | Matched component norm | 0.340° | 0.333° | 0.291° | 0.281° | The learned $A$ component produces much less rotation under RPSFT, while cross components have similar responses at equal norms. This matches the intended mechanism: selectively anchor $A$ while retaining cross-block adaptation. “Rotation-preserving” refers to this empirical reduction. We will clarify at first use that it does not imply exact subspace invariance. Figure 2 motivates the choice of block: in the measured query projections, a leading block occupying roughly 5% of the SVD core captures roughly 20% of empirical gradient energy. Appendix C.5 analyzes the associated adaptation–retention trade-off under a local loss model. We updated these explanations into the revision.
AuthorsOpenReview ↗

Response (2/2)

W2/Q(a): Size of the gains and comparison with simpler regularization

We agree that gains vary and that RPSFT does not win every metric. This is the reason we evaluated cross models and benchmarks. The non-math results in Table 4 are larger: across three training seeds, RPSFT improves ID/OOD means by 2.05/3.69 points on CSQA and 0.96/5.30 points on ProofWriter. Our claim is an improved empirical adaptation–retention balance across the evaluated settings, with task-dependent gains.

For simpler regularization, Table 12 compares RPSFT with full-weight L2 anchoring to the pretrained model. Below is the complete strength sweep on Qwen2.5-7B/ProofWriter at step 500. Values are mean ± sample SD over three training seeds; OOD averages MMLU-Pro, SuperGPQA, and SafetyOOD.

λ\lambdaRPSFT IDRPSFT OODL2 IDL2 OOD
0.183.56 ± 2.7346.89 ± 0.5577.56 ± 2.0447.87 ± 1.24
0.285.14 ± 1.3347.04 ± 1.1674.14 ± 0.8448.16 ± 0.06
0.581.02 ± 1.1747.44 ± 0.4576.79 ± 1.9848.09 ± 1.22
286.08 ± 1.9047.34 ± 1.2675.76 ± 1.1046.57 ± 0.76
584.46 ± 1.9147.50 ± 0.5974.72 ± 0.5047.11 ± 1.24
1085.71 ± 1.4647.47 ± 0.5174.12 ± 0.6947.42 ± 0.26

RPSFT achieves substantially stronger adaptation(8% by average) while keeping comparable OOD performance(difference less than 1% by average).

Dear Reviewer vPGS,

The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.