Learning to Evolve: Enabling LLM Continual Learning via the Stable Anchor Dynamics

Continual learning seeks to enable sequential mastery of a stream of tasks, simultaneously securing competent performance on the newly introduced task while preserving the proficiency acquired on earlier ones. In the contemporary era of large language models (LLMs), the performance degradation on old tasks, i.e., catastrophic forgetting, is partially mitigated by the inherently massive parameter scale and the extensive pre-training corpora. Nevertheless, due to the fundamental mechanics of gradient-based training, the acquisition of a new task invariably leaves an imprint on the parameter configuration, thereby exerting a nontrivial impact on previously established capabilities. In this paper, we probe into the very nature of forgetting and introduce the notion of Stable Anchor, which theoretically traces forgetting to the propagation of stable anchor changes from the new task to the old one (via the parameter space), ultimately altering its performance. Building upon this insight, we propose a loss function designed to minimize the influence transmitted via the stable anchor. Extensive experiments across diverse task streams substantiate the effectiveness of the proposed method.