Stabilizing Language Models under Continual Learning via Condition-Anchored Distillation

Continual adaptation of language models can change their output distribution on prompts learned earlier, while retaining every old prompt–answer pair may be undesirable or impossible. We study condition-anchored generative distillation (\CAGD): retain a small set of old prompts, use a frozen previous model to reconstruct completions and generation states, and match its predictive distributions while learning the next task. The formulation separates three roles that ordinary replay conflates: conditions select the behavior to protect, teacher generations locate relevant states, and soft targets specify how predictions may change. For autoregressive language generation, teacher-rollout distillation admits an exact chain-rule decomposition of sequence divergence. For masked-diffusion language modeling, our implementation directly controls local denoising drift on teacher-generated completions. In continual adaptation of a 219M masked diffusion language model, reduces four-task final held-out loss from 2.927 to 1.114 in one task order and from 2.168 to 0.891 in exact reverse. The same soft targets, in matched Qwen3-0.6B and 1.7B interventions, improve mean final GSM8K exact match by 3.36 and 1.82 percentage points and conditional retention by 11.38 and 4.97 points over hard replay. The direction persists on fresh facts and natural instructions across SMDM and Qwen3. In a longer two-boundary stream, Qwen answer-format retention improves. Together with an eight-task TRACE comparison, these results support condition-anchored functional preservation across the tested language-generation objectives. Code is available.