Preserve More, Freeze Less: Loss-Aware Subspaces for Continual Learning

Continually fine-tuning a pretrained language model requires acquiring new capabilities (plasticity) while retaining both its initial capabilities and those learned along the way (stability). Projection-based methods constrain parameter updates to protect directions associated with prior knowledge. Existing approaches select directions without measuring their importance for prior behavior, protect directions too coarsely, reducing plasticity, or store reference gradients, which does not scale to large models. We formulate an explicit preservation objective and derive a tractable approximation that identifies pairs of input and output directions in each linear transformation and estimates their importance from limited reference data. Building on this formulation, we develop SCLL, a projection-based continual learning method that protects only the most important pairs, improving the balance between preservation and adaptation. We evaluate SCLL on two continual learning benchmarks while protecting five initial capabilities. On TRACE, whose tasks differ in language, domain, and output format, SCLL improves average accuracy over the strongest projection baseline by 4 points while matching its retention and freezing fewer update directions; on Long CL, whose tasks require few new directions, it matches the strongest baseline. Protecting the selected initial capabilities also retains performance on related datasets not used for protection.