Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Large language models (LLMs) acquire new skills through fine-tuning, but fine-tuning can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent work connects forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid the principal directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. We introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting the top 99% of singular directions on both sides of each weight matrix still leaves enough low-energy capacity to learn new tasks, without task-specific subspace estimation. Our experiments show that SST improves in-domain performance while retaining prior capabilities. This behavior scales with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing $2$-$4\times$ less KL drift than the baseline recipes.