Understanding Forgetting Dynamics in Full Fine-Tuning of LLMs: A Neural Tangent Kernel Perspective

Full fine-tuning can cause large language models to forget previously mastered knowledge. Understanding these forgetting dynamics is essential for balancing adaptation and retention. Existing analyses based on neural tangent kernel (NTK) theory commonly assume a fixed gradient kernel. Our study shows that during full fine-tuning, the gradient kernel evolves primarily through a rapid decay in norm while retaining a moderately stable relative structure. Kernel evolution affects predictions under the NTK framework by substantially altering the response magnitude while having a smaller effect on its direction. To investigate how the direction-aligned but magnitude-biased NTK-based approximation affects parameter optimization, we introduce and compare two Pontryagin's maximum principle (PMP)-based parameter optimization methods, PG-NoDe and PG-De. Results show that, in our setting, a direction-aligned but magnitude-biased approximation NTK-based approximation can guide replay-weight optimization toward effective schedules. Calibrating the absolute prediction magnitudes provides limited additional benefit.