acceptodds
Under review as a conference paper at ICLR 2027

Understanding Forgetting Dynamics in Full Fine-Tuning of LLMs: A Neural Tangent Kernel Perspective

Abstract

Full fine-tuning can cause large language models to forget previously mastered knowledge. Understanding these forgetting dynamics is essential for balancing adaptation and retention. Existing analyses based on neural tangent kernel (NTK) theory commonly assume a fixed gradient kernel. Our study shows that during full fine-tuning, the gradient kernel evolves primarily through a rapid decay in norm while retaining a moderately stable relative structure. Kernel evolution affects predictions under the NTK framework by substantially altering the response magnitude while having a smaller effect on its direction. To investigate how the direction-aligned but magnitude-biased NTK-based approximation affects parameter optimization, we introduce and compare two Pontryagin's maximum principle (PMP)-based parameter optimization methods, PG-NoDe and PG-De. Results show that, in our setting, a direction-aligned but magnitude-biased approximation NTK-based approximation can guide replay-weight optimization toward effective schedules. Calibrating the absolute prediction magnitudes provides limited additional benefit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Full Fine-Tuning, Forgetting, Neural Tangent Kernel, Gradient Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pH35OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how gradient-kernel evolution affects forgetting during full fine-tuning of LLMs. It separates changes in kernel scale from changes in response direction and develops a decay-calibrated NTK approximation for correct-answer log-odds drift. An adjoint-based optimizer then uses the surrogate to select epoch-wise replay weights. The experiments find that decay calibration improves prediction magnitude, while replay schedules obtained without calibration achieve comparable downstream performance in the evaluated control setting.

Critical strengths
  1. The three questions in the introduction connect coherently to kernel measurements, response prediction, and replay control. The main contribution is the empirical distinction between accurate prediction magnitudes and useful optimization guidance.
  2. Tables 2 and 5 evaluate nine models on three datasets. Section 4.1 separates parameter fitting from held-out evaluation and reports variability across three data partitions. Table 1 supports the direction–magnitude distinction: calibration substantially reduces the parallel residual while leaving directional agreement largely unchanged.
  3. Table 3 reports the highest mean accuracy for PG-NoDe on all three datasets while showing that O-LoRA achieves lower forgetting. Appendix F uses a shared replay buffer, and Table 7 accounts for kernel-construction overhead.
Critical weaknesses
  1. Appendix F describes OPGD through output-gradient projection without explaining whether the cited method’s GAM-based gradient-norm penalty is retained. This leaves an implementation ambiguity rather than establishing an incorrect implementation. Selected values and tuning procedures for other baseline hyperparameters, including EWC and DER++, are also unspecified.
  2. The replay-control conclusion has narrower evidence than the prediction analysis: Table 3 evaluates only Qwen 1.5B, and Appendix B uses SGD throughout. The finding that magnitude calibration adds little control benefit should therefore remain explicitly scoped to this setting.
  3. The empirical remainder in Table 1 is estimated as observed log-odds change minus the prediction, as stated at lines 184–185. It can therefore include errors from the first-order dynamics, residual linearization, and continuous-time approximation, in addition to kernel evolution beyond shared scaling. Its attribution specifically to kernel evolution needs qualification.
Questions for authors
  1. Does OPGD include GAM and its gradient-norm penalty? Please provide baseline configurations.
  2. How should the empirical remainder be interpreted given the preceding approximation errors?
  3. Are additional replay-control results already available? Otherwise, please maintain the stated scope of the control conclusion.
5 comments

Dear Reviewer:

We sincerely thank you for your constructive feedback and insightful comments. In response, we have clarified the baseline implementation details, refined our interpretation of the empirical remainder, and conducted additional replay-control experiments.

Question 1: Does OPGD include GAM and its gradient-norm penalty? Please provide baseline configurations.

Response 1: Thank you for pointing out the ambiguity in our description of OPGD. Our OPGD implementation includes GAM and its gradient-norm penalty. Following the original OPGD paper (Zheng et al., 2026), we set the penalty strength to α=0.5\alpha=0.5 and the perturbation radius to b=0.05b=0.05. For EWC (Kirkpatrick et al., 2016), we use λ=5000\lambda=5000. For DER++ (Buzzega et al., 2020), we use (α,β)=(0.5,1)(\alpha,\beta)=(0.5,1). These method-specific configurations were selected with reference to the respective original papers.

To examine hyperparameter sensitivity, we conducted additional experiments. For OPGD, we varied b∈0.02,0.05,0.2b\in\\{0.02,0.05,0.2\\}. For EWC, we evaluated λ∈500,5000,50000\lambda\in\\{500,5000,50000\\}. For DER++, we evaluated (α,β)∈(0.5,1),(1,0.5),(0.75,0.75)(\alpha,\beta)\in\\{(0.5,1),(1,0.5),(0.75,0.75)\\}. Even the best configurations remain substantially behind PG-NoDe and PG-De in the overall accuracy–forgetting trade-off.

We have revised Appendix F to explicitly document GAM in OPGD and the hyperparameter values used for baselines, and added the sensitivity results in Tables 8-10.

Rebuttal Table 1. OPGD hyperparameter sensitivity (Accuracy (%) / Forgetting (%)).

Perturbation radius bbMMLUMedQACSQA
0.0249.16 / 36.7734.05 / 53.0767.12 / 20.99
0.0548.78 / 37.6333.56 / 54.5067.12 / 21.04
0.2048.20 / 39.0833.69 / 53.9067.46 / 20.45

Rebuttal Table 2. EWC hyperparameter sensitivity (Accuracy (%) / Forgetting (%)).

Regularization coefficient λ\lambdaMMLUMedQACSQA
50052.91 / 29.2836.08 / 47.3168.15 / 19.10
500053.51 / 27.7537.10 / 44.6667.70 / 19.32
5000053.52 / 26.1936.22 / 40.8368.11 / 18.33

Rebuttal Table 3. DER++ hyperparameter sensitivity (Accuracy (%) / Forgetting (%)).

Hyperparameter (α/β)(\alpha/\beta)MMLUMedQACSQA
0.50/1.0056.27 / 21.5141.72 / 29.8371.13 / 14.05
1.00/0.5056.12 / 20.6841.19 / 30.4371.33 / 13.33
0.75/0.7556.34 / 20.6841.58 / 29.8371.09 / 13.78

References

[1] Guodong Zheng, Peng Wang, Shengchao Hu, Quan Zheng, and Li Shen. Understanding the dynamics of forgetting and generalization in continual learning via the neural tangent kernel. In International Conference on Learning Representations, 2026.

[2] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. arXiv preprint arXiv:1612.00796, 2016.

[3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems, volume 33, 2020.

Question 2: How should the empirical remainder be interpreted given the preceding approximation errors?

Response 2: We appreciate this important observation and agree that the empirical remainder should not be exclusively attributed to kernel evolution. To examine how the preceding approximations affect the empirical remainder, we conducted additional experiments on MMLU, MedQA, and CSQA.

Replacing the linearized CE residual with its nonlinear form reduces the parallel remainder on all three datasets, with little change in the orthogonal remainder or directional agreement. Moreover, residual linearization produces a smaller change in the parallel component than that observed for kernel evolution in Table 1 of the manuscript.

Rebuttal Table 4. Effect of CE residual linearization.

DatasetPredictor∥m∥2\|m\|_2∥R∥∥2\|R_{\parallel}\|_2∥R⊥∥2\|R_{\perp}\|_2cos⁡(m,m+R)\cos(m,m+R)
MMLULinear237.62114.8619.790.9873
MMLUNonlinear202.0679.2919.730.9873
MedQALinear119.3572.568.110.9853
MedQANonlinear102.8456.088.240.9848
CSQALinear383.65212.0033.340.9817
CSQANonlinear286.86115.1132.840.9822

Replacing the continuous solution with discrete recurrence has a small effect on both the parallel and orthogonal remainders.

Rebuttal Table 5. Continuous versus discrete dynamics.

DatasetPredictor∥m∥2\|m\|_2∥R∥∥2\|R_{\parallel}\|_2∥R⊥∥2\|R_{\perp}\|_2cos⁡(m,m+R)\cos(m,m+R)
MMLUContinuous237.62114.8619.790.9873
MMLUDiscrete241.22118.4619.850.9872
MedQAContinuous119.3572.568.110.9853
MedQADiscrete121.5574.768.080.9854
CSQAContinuous383.65212.0033.340.9817
CSQADiscrete389.70218.0733.470.9815

Explicitly incorporating the full higher-order Taylor expansion is impractical in our setting. We therefore examine the remaining discrepancy after incorporating kernel decay, nonlinear residuals, and discrete recurrence. These changes substantially reduce the parallel remainder, while the orthogonal remainder remains largely unchanged. This persistent orthogonal remainder may partly reflect higher-order Taylor terms omitted by the first-order approximation.

Rebuttal Table 6. Jointly incorporating kernel decay, nonlinear residuals, and discrete recurrence.

DatasetPredictor∥m∥2\|m\|_2∥R∥∥2\|R_{\parallel}\|_2∥R⊥∥2\|R_{\perp}\|_2cos⁡(m,m+R)\cos(m,m+R)
MMLUNo decay + Linear + Continuous237.62114.8619.790.9873
MMLUDecay + Nonlinear + Discrete123.030.3320.240.9867
MedQANo decay + Linear + Continuous119.3572.568.110.9853
MedQADecay + Nonlinear + Discrete67.6420.817.860.9862
CSQANo decay + Linear + Continuous383.65212.0033.340.9817
CSQADecay + Nonlinear + Discrete154.0317.3934.540.9803

We fully agree that the empirical remainder cannot be attributed exclusively to kernel evolution. Accordingly, in Section 3.2, we replaced “two remainders capturing the effects of kernel evolution beyond this shared scaling” with “two remainder components describing the deviation from this prediction.” We have also added the corresponding analyses to Appendix L.

Question 3: Are additional replay-control results available? Otherwise, please maintain the stated scope of the control conclusion?

Response 3: We have conducted three additional sets of replay-control experiments beyond the original Qwen 1.5B setting, including experiments with AdamW, a 5× larger learning rate, and an additional evaluation on LLaMA 8B.

As shown in Rebuttal Tables 4-6, PG-NoDe and PG-De maintain strong performance across all three additional settings. In particular, one of the two PG variants achieves the highest accuracy across all nine dataset-setting combinations. These additional experiments provide further evidence that our proposed replay-control approach generalizes beyond the original Qwen 1.5B model, optimizer, and training configuration.

We have incorporated these additional replay-control results into Tables 14-16 of the revised manuscript. We have also added the following sentence at the end of Section 4.4: “Additional replay-control comparisons with AdamW, a 5× larger learning rate, and LLaMA 8B support the same conclusion (Appendix I).”

Meanwhile, we agree that this conclusion should be carefully scoped. Accordingly, we have revised the manuscript to emphasize replay control as a methodological contribution motivated by our kernel analysis, rather than a general finding. Specifically, we have removed statements suggesting that “magnitude calibration offers limited additional benefit” from the Abstract, Introduction, and other related sections.

Rebuttal Table 7. Method comparison under AdamW (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM44.66 / 46.2426.23 / 75.9356.93 / 36.94
OPGD42.35 / 51.8230.55 / 70.0222.07 / 88.38
EWC44.23 / 48.5529.44 / 68.1322.65 / 88.06
O-LoRA57.48 / 0.9143.06 / 1.3675.95 / 0.23
BLF43.69 / 49.3028.62 / 70.1021.86 / 89.37
ER53.10 / 33.0433.69 / 60.5645.40 / 55.63
ER+DWA53.13 / 33.0033.79 / 60.8646.53 / 54.19
DER++54.91 / 29.9445.68 / 28.2448.96 / 52.03
Smart Replay56.10 / 25.8146.07 / 27.0257.48 / 37.61
PG-NoDe59.57 / 17.3347.35 / 22.1878.00 / 6.98
PG-De58.98 / 20.4347.25 / 23.7778.07 / 6.98

Rebuttal Table 8. Method comparison with a 5× larger learning rate (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM48.32 / 37.7631.73 / 59.6568.39 / 19.19
OPGD46.09 / 43.0131.20 / 61.0967.43 / 21.26
EWC49.96 / 35.1933.30 / 55.5666.92 / 21.40
O-LoRA57.65 / 2.5243.39 / 3.9475.47 / 1.26
BLF48.11 / 39.2931.76 / 59.2766.47 / 21.98
ER53.86 / 27.3035.66 / 48.9071.40 / 15.00
ER+DWA53.84 / 27.3035.99 / 48.2271.33 / 15.00
DER++55.84 / 22.7541.00 / 32.9372.25 / 13.38
Smart Replay56.29 / 20.8042.37 / 21.7371.40 / 14.95
PG-NoDe58.88 / 13.2844.53 / 10.8377.45 / 6.49
PG-De58.81 / 13.7344.99 / 11.8875.95 / 7.48

Rebuttal Table 9. Method comparison on LLaMA 8B (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM53.06 / 36.6868.63 / 20.0467.57 / 24.31
OPGD54.17 / 35.1867.13 / 22.9565.96 / 26.48
EWC53.08 / 37.0460.71 / 27.3865.96 / 25.85
O-LoRA60.98 / 0.8365.29 / 1.3376.12 / 0.27
BLF52.56 / 37.7559.00 / 29.5365.38 / 26.52
ER60.50 / 25.1069.84 / 19.6870.00 / 21.02
ER+DWA60.52 / 24.9869.78 / 19.6869.93 / 21.02
DER++61.21 / 23.8369.81 / 19.0770.37 / 20.25
Smart Replay61.69 / 22.9272.27 / 15.5571.88 / 18.27
PG-NoDe67.23 / 15.0671.94 / 15.4078.28 / 9.11
PG-De65.11 / 18.1472.63 / 13.7777.35 / 10.10

We sincerely thank you again for highlighting these important issues. Your suggestions have helped us improve the clarity of our implementation details, the rigor of our empirical interpretation, and the scope of our experimental conclusions. We sincerely look forward to your further feedback.

Reviewer_iShKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using a gradient kernel for understanding fine-tuning and forgetting dynamics in LLM training. The paper proposes 2 replay-weight optimization methods for modeling the kernel based on this approach to select replay data weights. The results show that this can offer some benefits for mitigating forgetting using a replay buffer

Critical strengths
  • The paper provides empirical results that show gradient kernel can still provide useful information about forgetting dynamics even though the gradient changes quite a bit during training.
  • The weighted replay dataset experiment results show reduced forgetting using the prediction based on the imperfect approximation of the training dynamics. This is a good example of the application of the previous point's empirical results.
Critical weaknesses
  • The epoch-level experiments with varying weights do not necessarily establish this as the most optimal strategy. For instance, does it outperform a well-tuned constant weight across epochs?
  • The replay weighting experiment results are on a single model. Although Table 2 shows the correlation of predicted score and observed score for various models, the main results don't show if different model families/sizes benefit from the weight replay schedules
Questions for authors
  • Could authors provide results for more model families or justification for the choice of the model as mentioned in the weakness section?
  • Could the authors provide additional results for constant weight tunning experiment at the epoch level mentioned in the weakness section?
5 comments

Dear Reviewer:

We sincerely thank you for your constructive feedback and valuable suggestions. In response to your comments, we have made the following revisions and conducted additional experiments.


Question 1: Could the authors provide results for more model families or justify the choice of the model as mentioned in the weakness section?

Response 1: To investigate whether our replay-weight optimization approach generalizes beyond the original Qwen model, we extended the replay-control experiments to LLaMA 8B, representing an additional model family and a larger model scale. We evaluated PG-NoDe and PG-De against the same baselines on MMLU, MedQA, and CSQA.

As shown in Rebuttal Table 1, one of the two PG variants achieves the highest accuracy across all three datasets. These results provide additional evidence that our replay-weight optimization approach generalizes beyond the original Qwen model. Furthermore, comparing PG-NoDe with PG-De shows that incorporating gradient decay provides limited additional performance benefits, with neither variant consistently outperforming the other across all three datasets.

We have incorporated the additional LLaMA 8B replay-control results into Table 16 of the revised manuscript. We have also added the following sentence at the end of Section 4.3: "Additional replay-control comparisons with AdamW, a 5× larger learning rate, and Llama 8B support the same conclusion (Appendix I)."

Rebuttal Table 1. Method comparison on LLaMA 8B (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM53.06 / 36.6868.63 / 20.0467.57 / 24.31
OPGD54.17 / 35.1867.13 / 22.9565.96 / 26.48
EWC53.08 / 37.0460.71 / 27.3865.96 / 25.85
O-LoRA60.98 / 0.8365.29 / 1.3376.12 / 0.27
BLF52.56 / 37.7559.00 / 29.5365.38 / 26.52
ER60.50 / 25.1069.84 / 19.6870.00 / 21.02
ER+DWA60.52 / 24.9869.78 / 19.6869.93 / 21.02
DER++61.21 / 23.8369.81 / 19.0770.37 / 20.25
Smart Replay61.69 / 22.9272.27 / 15.5571.88 / 18.27
PG-NoDe67.23 / 15.0671.94 / 15.4078.28 / 9.11
PG-De65.11 / 18.1472.63 / 13.7777.35 / 10.10

Question 2: Could the authors provide additional results for a constant-weight tuning experiment at the epoch level mentioned in the weakness section?

Response 2: To provide a direct comparison between dynamic and constant replay-weight strategies, we introduced an additional constraint into the PMP optimization procedure, requiring the replay weight to remain identical across all epochs (w1=w2=⋯=wTw_1=w_2=\cdots=w_T).

We then compared the epoch-wise replay-weight controller against the constant-weight baseline under the same evaluation protocol. Each result is reported as accuracy (%) / forgetting (FT, %). As shown in Rebuttal Table 2, the epoch-wise controller consistently achieves higher accuracy and lower forgetting across all three datasets. These results demonstrate the advantage of dynamically adapting replay weights across epochs over using a single constant replay weight.

We have incorporated these additional results into Table 17 in Appendix J and added a corresponding discussion titled "Epoch-wise versus constant replay weights" to Section 4.4.

Rebuttal Table 2. Comparison between epoch-wise and constant replay weights (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
Epoch-wise replay weights59.81 / 9.4046.27 / 11.2877.09 / 5.14
Constant replay weight57.88 / 15.5943.65 / 15.0673.59 / 9.86

We sincerely appreciate your valuable comments and suggestions, which have helped improve our manuscript. Please do not hesitate to contact us if you have further concerns.

Hi, thank you for providing the results. Was the constant replay weight selected solely by optimizing the surrogate objective, or was it also tuned using actual validation performance?

Dear Reviewer:

We would like to clarify that the constant replay weight was tuned using actual validation performance. Specifically, we followed the same tuning procedure used for the PG-Replay results in Table 6. The only difference was that we constrained the replay weights to be identical across all epochs, ensuring comparability between the constant-weight and epoch-wise-weight settings.

Thank you for your follow-up question. We sincerely look forward to your further feedback.

Reviewer_m9J2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how log-odds gradient kernels change during full fine-tuning. It finds that kernel scale decays more strongly than the relative response pattern changes. A finite-time approximation with fitted decay improves prediction magnitudes; a replay-control study then finds that this calibration adds little over a fixed-kernel surrogate.

The introduction’s three questions are followed through the method, experiments and conclusion with particularly good thematic consistency.

Critical strengths

The distinction between response direction and magnitude is useful and testable. The paper reports both correlation and RMSE, making the consequences of magnitude calibration visible.

The response study spans nine models. Appendix E separates fitting and evaluation along both sample and time axes: checkpoints 0–2 supply calibration, followed by later held-out prediction. This is a meaningful experimental strength.

The paper also reports setup costs and clearly qualifies the empirical affine-bound analysis. Its writing generally distinguishes an observed phenomenon from a broader theoretical guarantee.

Critical weaknesses

The main unresolved issue is whether the surrogate supplies reliable control gradients for actual training. Equation (22) treats the kernel trajectory as prescribed while replay weights vary. In the target model, changing those weights changes the parameter trajectory and potentially the kernel itself. Alignment between two surrogate gradients, as in Table 4, does not validate either against the actual training objective’s gradient.

The response study is broader than the control study. Replay-control results are reported only for Qwen 1.5B on three multiple-choice benchmarks, using a common small-learning-rate SGD protocol. The conclusion about magnitude calibration’s limited value therefore remains conditional on a narrow optimization regime. A strong tuned constant replay-weight baseline would help establish whether epoch-wise control is necessary.

The efficiency claim also requires careful interpretation. Table 7 supports earlier crossing of Smart Replay’s best accuracy, but PG-NoDe takes longer to complete the full run. The sampled local affine analysis does not certify the implemented constrained optimizer, a distinction the authors appropriately acknowledge.

Questions for authors
  1. How well do surrogate schedule gradients agree with finite-difference changes in the actual fine-tuning objective, including under substantially different replay schedules?

  2. Does the controller outperform a tuned constant replay weight at matched computation and acquisition–retention operating points?

  3. Do the scale-versus-direction finding and PG-NoDe/PG-De comparison persist with AdamW, larger learning rates and another model family in the control experiment?

3 comments

Dear Reviewer:

We sincerely thank you for your constructive feedback and valuable suggestions. In response to your comments, we have made the following revisions and conducted additional experiments.

Question 1: How well do surrogate schedule gradients agree with finite-difference changes in the actual fine-tuning objective, including under substantially different replay schedules?

Response 1: To further validate the reliability of our surrogate schedule gradients, we computed forward finite-difference gradients using a +0.5 perturbation at five representative points along the optimization trajectory: the starting point, 1/4 point, midpoint, 3/4 point, and endpoint. Specifically, we approximated the actual objective gradient at each point as:

gFD(w)=L(w+0.5)−L(w)0.5.g_{\mathrm{FD}}(w)=\frac{\mathcal{L}(w+0.5)-\mathcal{L}(w)}{0.5}.

We then measured the cosine similarity between these finite-difference gradients and the corresponding surrogate schedule gradients. As reported in Rebuttal Table 1, the overall average cosine similarity across all datasets and optimization stages is 0.9157, demonstrating strong directional agreement between the surrogate schedule gradients and the actual finite-difference gradients.

Rebuttal Table 1. Cosine similarity between surrogate schedule gradients and forward finite-difference gradients at five points along the optimization trajectory.

DatasetStarting point1/4 pointMidpoint3/4 pointEndpoint
MMLU0.98990.83200.83870.83550.8392
MedQA0.78560.85490.98140.97400.9742
CSQA0.89110.98140.98830.98780.9812

Question 2: Does the controller outperform a tuned constant replay weight at matched computation and acquisition–retention operating points?

Response 2: To provide a direct comparison between dynamic and constant replay-weight strategies, we introduced an additional constraint into the PMP optimization procedure, requiring the replay weight to remain identical across all epochs (w1=w2=⋯=wTw_1=w_2=\cdots=w_T).

We then compared the epoch-wise replay-weight controller against the constant-weight baseline under the same evaluation protocol. Each result is reported as accuracy (%) / forgetting (FT, %). The epoch-wise controller consistently achieves higher accuracy and lower forgetting across all three datasets. These results demonstrate the advantage of dynamically adapting replay weights across epochs over using a single constant replay weight.

We have added this comparison to Section 4.4, noting that epoch-wise replay weights achieve higher accuracy and lower forgetting than the optimized constant-weight baseline on all three datasets. The experimental details and results are provided in Appendix J, Table 17.

Rebuttal Table 2. Comparison between epoch-wise and constant replay weights (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
Epoch-wise replay weights59.81 / 9.4046.27 / 11.2877.09 / 5.14
Constant replay weight57.88 / 15.5943.65 / 15.0673.59 / 9.86

Question 3: Do the scale-versus-direction finding and PG-NoDe/PG-De comparison persist with AdamW, larger learning rates and another model family in the control experiment?

Response 3: To investigate the generality of our approach, we conducted three additional experiments using AdamW, a 5× larger learning rate, and LLaMA 8B.

We first examine whether the scale-versus-direction finding generalizes to these settings. The cosine remains high across different setting. Incorporating kernel decay substantially changes the parallel remainder, whereas the orthogonal remainder remains stable.

Rebuttal Table 3. Gradient-kernel analysis under AdamW.

DatasetPredictor∥m∥2\|m\|_2∥R∥∥2\|R_{\parallel}\|_2∥R⊥∥2\|R_{\perp}\|_2cos⁡(m,m+R)\cos(m,m+R)
MMLUNo decay1206.51030.846.40.96
MMLUDecay298.9124.852.10.95
MedQANo decay380.8326.810.90.98
MedQADecay125.571.812.50.97
CSQANo decay2874.72476.280.40.98
CSQADecay649.2251.986.60.97

Rebuttal Table 4. Gradient-kernel analysis with a 5× larger learning rate.

DatasetPredictor∥m∥2\|m\|_2∥R∥∥2\|R_{\parallel}\|_2∥R⊥∥2\|R_{\perp}\|_2cos⁡(m,m+R)\cos(m,m+R)
MMLUNo decay84.049.26.00.98
MMLUDecay64.429.66.00.98
MedQANo decay16.56.43.90.93
MedQADecay13.53.64.50.90
CSQANo decay79.250.88.00.96
CSQADecay46.1917.797.840.96

Rebuttal Table 5. Gradient-kernel analysis on LLaMA 8B.

DatasetPredictor∥m∥2\|m\|_2∥R∥∥2\|R_{\parallel}\|_2∥R⊥∥2\|R_{\perp}\|_2cos⁡(m,m+R)\cos(m,m+R)
MMLUNo decay116.959.26.20.99
MMLUDecay66.28.66.50.99
MedQANo decay58.043.65.80.92
MedQADecay40.726.35.80.92
CSQANo decay142.585.06.00.99
CSQADecay71.213.76.30.99

We further examine whether the performance of our methods generalizes to these settings. PG-NoDe and PG-De achieve strong performance, with one variant attaining the highest accuracy. Their comparable results suggest that kernel decay provides limited additional benefits.

Rebuttal Table 6. Method comparison under AdamW (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM44.66 / 46.2426.23 / 75.9356.93 / 36.94
OPGD42.35 / 51.8230.55 / 70.0222.07 / 88.38
EWC44.23 / 48.5529.44 / 68.1322.65 / 88.06
O-LoRA57.48 / 0.9143.06 / 1.3675.95 / 0.23
BLF43.69 / 49.3028.62 / 70.1021.86 / 89.37
ER53.10 / 33.0433.69 / 60.5645.40 / 55.63
ER+DWA53.13 / 33.0033.79 / 60.8646.53 / 54.19
DER++54.91 / 29.9445.68 / 28.2448.96 / 52.03
Smart Replay56.10 / 25.8146.07 / 27.0257.48 / 37.61
PG-NoDe59.57 / 17.3347.35 / 22.1878.00 / 6.98
PG-De58.98 / 20.4347.25 / 23.7778.07 / 6.98

Rebuttal Table 7. Method comparison with a 5× larger learning rate (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM48.32 / 37.7631.73 / 59.6568.39 / 19.19
OPGD46.09 / 43.0131.20 / 61.0967.43 / 21.26
EWC49.96 / 35.1933.30 / 55.5666.92 / 21.40
O-LoRA57.65 / 2.5243.39 / 3.9475.47 / 1.26
BLF48.11 / 39.2931.76 / 59.2766.47 / 21.98
ER53.86 / 27.3035.66 / 48.9071.40 / 15.00
ER+DWA53.84 / 27.3035.99 / 48.2271.33 / 15.00
DER++55.84 / 22.7541.00 / 32.9372.25 / 13.38
Smart Replay56.29 / 20.8042.37 / 21.7371.40 / 14.95
PG-NoDe58.88 / 13.2844.53 / 10.8377.45 / 6.49
PG-De58.81 / 13.7344.99 / 11.8875.95 / 7.48

Rebuttal Table 8. Method comparison on LLaMA 8B (Accuracy (%) / Forgetting (%)).

MethodMMLUMedQACSQA
GPM53.06 / 36.6868.63 / 20.0467.57 / 24.31
OPGD54.17 / 35.1867.13 / 22.9565.96 / 26.48
EWC53.08 / 37.0460.71 / 27.3865.96 / 25.85
O-LoRA60.98 / 0.8365.29 / 1.3376.12 / 0.27
BLF52.56 / 37.7559.00 / 29.5365.38 / 26.52
ER60.50 / 25.1069.84 / 19.6870.00 / 21.02
ER+DWA60.52 / 24.9869.78 / 19.6869.93 / 21.02
DER++61.21 / 23.8369.81 / 19.0770.37 / 20.25
Smart Replay61.69 / 22.9272.27 / 15.5571.88 / 18.27
PG-NoDe67.23 / 15.0671.94 / 15.4078.28 / 9.11
PG-De65.11 / 18.1472.63 / 13.7777.35 / 10.10

We added the gradient-kernel analyses to Appendix H, and the corresponding replay-control results to Appendix I. Sections 4.3 and 4.4 now briefly summarize the consistent findings across these settings.

We sincerely thank you again for your constructive feedback. We look forward to your further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.