Understanding and Mitigating Forgetting in LLM Full Fine-Tuning at the Parameter Level

Full fine-tuning enables large language models to acquire new knowledge but can cause them to forget previously mastered knowledge. Existing gradient-projection methods constrain overall update directions without explicitly identifying which parameter updates contribute to forgetting. We decompose gradient similarity into parameter-wise contributions to distinguish Conflicting Parameters, whose updates increase the loss on mastered examples to first order, from Collaborative Parameters, whose updates do not. Based on this analysis, we propose Collaborative Parameter Learning (CPL), which uses a gradient from reference examples to block conflicting updates while retaining collaborative ones. Experiments span five models and six datasets across three modalities. Across held-out evaluations, CPL compares favorably with existing forgetting-mitigation baselines, achieving a strong balance between accuracy and knowledge retention. Further analyses suggest that CPL’s strong performance is likely related to the recovery of previously mastered answers and greater preservation of upper-layer representation structure.