A New Forgetting-Risk Constrained Optimization for Model Fine-Tuning

Fine-tuning a large pre-trained model is often necessary for strong downstream performance, but can degrade generalization and cause catastrophic forgetting of pre-trained knowledge. Existing forgetting-mitigation methods commonly regularize parameter changes or control selected spectral structures, such as suppressing intruder dimensions or protecting top singular components. Such component-wise control may provide limited spectral coverage, as unprotected directions can become prominent during adaptation and contribute to forgetting. Motivated by the observation that unconstrained adaptation can develop prominent task-specific spectral directions associated with forgetting, we regularize directional flexibility by restricting updates to an admissible subspace. We formulate forgetting-aware fine-tuning with joint adaptation-subspace and update-magnitude constraints, using the dominant column subspace of the pre-trained weights as a data-free, model-informed choice. We develop two realizations: Representation-Space Projected LoRA (RSP-LoRA), which projects LoRA updates onto the selected subspace, and Vector-based Feature Adaptation (VeFA), which satisfies the constraint through feature-side parameterization. Across image classification, natural language understanding, and Qwen2.5-7B mathematical reasoning, our methods substantially mitigate forgetting while maintaining competitive downstream performance. Ablations show that subspace restriction itself reduces forgetting, while the dominant pre-trained subspace generally provides a more favorable stability–plasticity trade-off than dimension-matched random and bottom subspaces.