Rotation-Preserving Supervised Fine-Tuning

Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work relates this degradation to changes in dominant singular subspaces of pretrained weight matrices. Identifying dominant loss-sensitive subspaces from full Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose Rotation-Preserving Supervised Fine-Tuning (RPSFT), which anchors the dominant top-$k$ product block in the pretrained SVD basis, motivated by the concentration of empirical-Fisher gradient energy in leading pretrained coordinates. This block-level anchor empirically reduces dominant-subspace rotation while preserving task adaptation. Across model families and sizes, RPSFT improves the in-domain/OOD trade-off over standard SFT and strong SFT baselines on math and non-math adaptation, and provides stronger initializations for downstream RL fine-tuning.