acceptodds
Under review as a conference paper at ICLR 2027

A Unified NTK-based Framework for Forgetting Approximation in Full Fine-Tuning LLMs

Abstract

Full fine-tuning enables large language models (LLMs) to acquire new knowledge but also alters previously mastered behavior. Therefore, understanding how forgetting arises during fine-tuning is crucial. We develop a neural tangent kernel (NTK)-based framework to locally approximate output changes associated with forgetting. The derivation separates the change into Gauss–Newton (GGN) quadratic term and linearization remainder terms. We use GGN quadratic to approximate the output change, which consists of three factors: transfer geometry, curvature response, and residual signal. We further show that three major forgetting-mitigation approaches can be viewed as modifications to these factors. Based on this framework, we proposed NTK-Replay, which selects replay strength by balancing approximated drift against replay penalty. Extensive experiments covering three modalities, five datasets, and twelve models validate the effectiveness of our NTK-based approximation framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Full Fine-Tuning, Forgetting, Neural Tangent Kernel

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9ArZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies a second-order local expansion to cross-entropy and a first-order expansion to model outputs. It decomposes drift on the mastered set into transfer geometry, curvature response, and residual signal, then derives effective kernels for projection, quadratic regularization, and replay. NTK-Replay selects the replay weight by minimizing normalized predicted drift plus a penalty on replay strength. Experiments span text, image, and speech models, reporting drift correlations for 32 model–dataset combinations and replay results on three text datasets.

Critical strengths
  1. The unified formulation is somewhat novel. The framework explicitly incorporates cross-entropy curvature and describes projection, regularization, and replay through a common effective-kernel language. For the three mitigation strategies, reported drift correlations are approximately 0.92–0.96.
  2. The empirical scope and cost analysis are substantial. Drift prediction is evaluated across three modalities, five datasets, and twelve models. The appendices also report RMSE, damping sensitivity, sketching time, and memory use.
Critical weaknesses
  1. The theoretical predictor does not match the training dynamics. Equations (15)–(20) describe the stationary point of a damped local surrogate, while the experiments use ten epochs of SGD without weight decay; the damping coefficient is set only from the kernel spectral radius. High Pearson correlation does not establish scale accuracy, and some RMSE values exceed 3 log-odds units.
  2. The source of NTK-Replay's improvement is not isolated. The evaluation lacks a thoroughly tuned fixed replay weight, an oracle weight selected using observed validation drift, and random or grid-selected weight controls. The reported time-to-quality comparison also does not replace a comparison at equal total compute.
Questions for authors
  1. If the predictor is replaced by finite-step, undamped linearized SGD dynamics matching the experiments, what are the resulting correlation, RMSE, calibration slope, and intercept? Please also report Jacobian drift and update norms during training, and relate them to prediction error.
  2. With the buffer, total compute, and tuning budget held fixed, does NTK-Replay still outperform a well-tuned fixed-weight ER baseline, an oracle replay weight, and a simple grid search? Please report new-task learning, old-task retention, the selected replay weights, and total wall-clock time separately.
3 comments

Dear Reviewer,

We sincerely appreciate your thoughtful review and constructive suggestions. In response to your comments, we have conducted additional theoretical analyses and experiments, as detailed below.

Question 1: If the predictor is replaced by finite-step, undamped linearized SGD dynamics matching the experiments, what are the resulting correlation, RMSE, calibration slope, and intercept? Please also report Jacobian drift and update norms during training, and relate them to prediction error.

Response 1: To investigate the finite-step, undamped linearized SGD dynamics, we compared predictions obtained by recursively evolving the dynamics up to the 10th epoch with those derived from the stationary-point solution. As shown in Rebuttal Table 1, the two predictions yield nearly identical Pearson correlations, RMSEs, calibration slopes, and intercepts.

To understand this similarity, we further examined the cross-entropy loss at the end of each epoch in Rebuttal Table 2. The loss decreases progressively and approaches a plateau by the 10th epoch, suggesting that the models are already close to convergence. Consequently, the finite-step dynamics at the 10th epoch produce predictions that are nearly identical to those obtained from the stationary-point solution.

We have added the finite-step prediction results to Appendix H and updated Section 4.2 to state: “We study the effect of a finite prediction horizon in Appendix H.”

Rebuttal Table 1. Steady-state versus 10-epoch finite-step predictions.

DatasetResponsePearsonRMSECalibration slopeIntercept
MMLUSteady state0.7413.1400.744-2.589
MMLUFinite steps (10 epochs)0.7413.1400.744-2.589
MedQASteady state0.8102.1740.876-1.006
MedQAFinite steps (10 epochs)0.8102.1740.876-1.006
CSQASteady state0.6693.2770.701-3.242
CSQAFinite steps (10 epochs)0.6693.2770.702-3.243

Rebuttal Table 2. End-of-epoch training cross-entropy loss.

EpochMMLU lossMedQA lossCSQA loss
12.5685902.2801963.646660
21.9539451.7200382.489300
31.7180591.5648652.018899
41.6000951.5046281.780756
51.5327841.4739761.645739
61.4909611.4556781.562545
71.4632621.4435211.508311
81.4438861.4349141.470937
91.4297811.4284861.444199
101.4191771.4235071.424399

We additionally tracked the Jacobian drift and parameter update norms throughout training and report them alongside the prediction RMSE in Rebuttal Table 3. Both Jacobian drift and parameter update norms consistently increase with training epochs. Meanwhile, the prediction RMSE also increases progressively. Notably, the growth in Jacobian drift closely accompanies the increase in prediction error, suggesting that the evolving Jacobian contributes to the degradation of prediction accuracy under the fixed-Jacobian approximation. The increasing parameter update norms further indicate that the model progressively moves away from its initial parameter state, providing a plausible explanation for the observed Jacobian evolution.

Rebuttal Table 3. Epoch-wise RMSE, Jacobian drift, and update norms.

EpochMMLU RMSEMMLU Jacobian driftMMLU update normsMedQA RMSEMedQA Jacobian driftMedQA update normsCSQA RMSECSQA Jacobian driftCSQA update norms
11.42037187.40.0271.04829798.40.0270.99826071.00.012
22.00045472.80.0361.56036589.40.0371.84438025.70.019
32.33549499.90.0421.79939893.70.0422.34144854.40.024
42.56252026.70.0461.92641762.40.0462.65049256.40.028
52.72753805.10.0492.00642942.80.0482.85452395.00.031
62.85055141.10.0522.06043762.20.0502.99354766.30.033
72.94756187.90.0542.10044373.30.0523.09456612.30.035
83.02457033.70.0562.13144853.30.0533.17158076.30.036
93.08757733.80.0572.15445244.60.0543.23059255.50.038
103.14058324.80.0592.17445572.60.0553.27760222.00.039

Question 2: With the buffer, total compute, and tuning budget held fixed, does NTK-Replay still outperform a well-tuned fixed-weight ER baseline, an oracle replay weight, and a simple grid search? Please report new-task learning, old-task retention, the selected replay weights, and total wall-clock time separately.

Response 2: To evaluate the computational cost of tuning fixed replay weights, we conducted a progressive grid search for ER, starting from a replay weight of 1 and doubling it at each successive search round (ρ∈{1,2,4,8}\rho \in \{1, 2, 4, 8\}). We accumulated the wall-clock time across successive search rounds to reflect the actual tuning cost.

As shown in Rebuttal Table 4, after four search rounds, the cumulative wall-clock time for ER reaches 10.16 hours on MMLU, 12.48 hours on MedQA, and 3.32 hours on CSQA, compared with only 3.30, 4.04, and 1.31 hours for NTK-Replay, respectively. Given the high computational cost of LLM training, extensive hyperparameter searches are particularly time-consuming. Beyond this computational advantage, NTK-Replay also achieves higher accuracy across all three datasets.

We further conducted a hyperparameter search for DER++-based replay using three different ((\alpha,\beta)) configurations. As shown in Rebuttal Table 5, these configurations also require considerable cumulative training time, yet none matches the overall performance of NTK-Replay.

Overall, these results demonstrate that NTK-Replay can adaptively determine replay weights to achieve a more favorable acquisition-retention trade-off, while avoiding the substantial computational cost associated with repeatedly training models under different fixed-weight configurations.

The ER and DER++ search results are now reported in Appendix J. Section 4.4 states: “Appendix J investigates replay performance and cumulative tuning cost for ER and DER++.”

Rebuttal Table 4. Comparison of progressive fixed-weight ER search and NTK-Replay in terms of knowledge acquisition, forgetting, and cumulative computational cost. Each entry reports Accuracy (%) / Learning (%) / Forgetting (%) / Total wall-clock time (h).

DatasetMethodAccuracy (%) ↑Learning (%) ↑Forgetting (%) ↓Total wall-clock time (h) ↓
MMLUER (1st round, ρ=1\rho=1)54.8428.5226.092.54
MMLUER (2nd round, ρ=2\rho=2)56.9926.8121.155.08
MMLUER (3rd round, ρ=4\rho=4)57.8422.4316.517.62
MMLUER (4th round, ρ=8\rho=8)58.8018.6112.0910.16
MMLUNTK-Replay60.3517.628.693.30
MedQAER (1st round, ρ=1\rho=1)38.5822.1641.003.12
MedQAER (2nd round, ρ=2\rho=2)41.9918.4428.736.24
MedQAER (3rd round, ρ=4\rho=4)44.1812.8116.809.36
MedQAER (4th round, ρ=8\rho=8)45.449.7910.2012.48
MedQANTK-Replay46.4612.4911.244.04
CSQAER (1st round, ρ=1\rho=1)69.8124.6415.570.83
CSQAER (2nd round, ρ=2\rho=2)71.2223.3313.291.66
CSQAER (3rd round, ρ=4\rho=4)72.9620.7110.142.49
CSQAER (4th round, ρ=8\rho=8)74.4418.377.423.32
CSQANTK-Replay77.2822.314.921.31

Rebuttal Table 5. Progressive hyperparameter search for DER++ and comparison with NTK-Replay. Each entry reports Accuracy (%) / Learning (%) / Forgetting (%) / Total wall-clock time (h).

DatasetMethodAccuracy (%) ↑Learning (%) ↑Forgetting (%) ↓Total wall-clock time (h) ↓
MMLUDER++ (1st round, (α,β)=(0.50,10)(\alpha,\beta)=(0.50,10))56.2726.1921.513.75
MMLUDER++ (2nd round, (α,β)=(1.00,0.50)(\alpha,\beta)=(1.00,0.50))56.1224.7320.6810.90
MMLUDER++ (3rd round, (α,β)=(0.75,0.75)(\alpha,\beta)=(0.75,0.75))56.3425.2420.6818.21
MMLUNTK-Replay60.3517.628.693.30
MedQADER++ (1st round, (α,β)=(0.50,1.00)(\alpha,\beta)=(0.50,1.00))41.7220.0229.834.26
MedQADER++ (2nd round, (α,β)=(1.00,0.50)(\alpha,\beta)=(1.00,0.50))41.1919.5630.4312.31
MedQADER++ (3rd round, (α,β)=(0.75,0.75)(\alpha,\beta)=(0.75,0.75))41.5819.7929.8319.20
MedQANTK-Replay46.4612.4911.244.04
CSQADER++ (1st round, (α,β)=(0.50,1.00)(\alpha,\beta)=(0.50,1.00))71.1324.3214.051.44
CSQADER++ (2nd round, (α,β)=(1.00,0.50)(\alpha,\beta)=(1.00,0.50))71.3322.9013.332.79
CSQADER++ (3rd round, (α,β)=(0.75,0.75)(\alpha,\beta)=(0.75,0.75))71.0923.3313.784.16
CSQANTK-Replay77.2822.314.921.31

Question 3: High Pearson correlation does not establish scale accuracy, and some RMSE values exceed 3 log-odds units.

Response 3: We acknowledge that the RMSE exceeds 3 log-odds on some datasets. To further evaluate the scale accuracy of our predictor, we compared our full CE response with the NTK baseline from Liu et al. (TPAMI 2026). As shown in Rebuttal Table 6, our method achieves substantially lower RMSE across all three datasets, demonstrating considerably better prediction accuracy than the existing NTK-based approach.

Rebuttal Table 6. RMSE comparison between our full CE response and the NTK baseline from TPAMI 2026.

PredictorMMLUMedQACSQA
Full CE response3.1402.1743.277
TPAMI 20269.0377.16310.023

[1] J Liu et al. Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective. TPAMI, 2026.

We sincerely thank you for your time and valuable feedback. Your comments have encouraged us to further investigate the theoretical and empirical aspects of our approach. We will incorporate the additional analyses, experimental results, and corresponding clarifications into the revised manuscript. We look forward to your further feedback.

Reviewer_tzZPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives a local NTK/GGN approximation of log-odds drift on mastered examples during cross-entropy LLM full fine-tuning. The approximation decomposes into transfer geometry, curvature response and residual signal. It reinterprets projection, regularization and replay as modifications of these factors, and proposes NTK-Replay to select the replay weight. Predicted and observed drift correlate at 0.796 on average across 12 models and 3 modalities. NTK-Replay achieves the best accuracy on three text datasets.

Critical strengths
  1. The paper extends NTK forgetting analysis to cross-entropy LLM fine-tuning with broad empirical validation, and gives a unified local view of three mitigation families.

  2. NTK-Replay substantially reduces forgetting relative to replay baselines while keeping the highest accuracy, and FG-Sketch makes gradient-kernel computation practical.

Critical weaknesses

S1: For most samples, pF<0.1p_F<0.1, making HFH_F small and 1−pF≈11-p_F\approx 1. As a result, the proposed predictor is close to KMF1/μK_{MF}\mathbf{1}/\mu. The shuffle experiments also show little impact from these terms. Comparisons with the squared-loss NTK formulation and simpler gradient-similarity baselines would help clarify the contribution. Moreover, the very small learning rate (2e-7) may limit the conclusions to the lazy-training regime.

S2: An important comparison with ER using a tuned ρ\rho is missing. The method also appears sensitive to ω\omega, with CSQA FT results ranging from 4.25 to 8.74, while requiring 1.4–2× the computational cost of Smart Replay. In addition, the model used in Tables 3–4 should be clearly specified.

Questions for authors
  1. How does the predictor compare against squared-loss NTK and KMF1K_{MF}\mathbf{1}?

  2. Do the correlations hold at standard learning rates or with Adam?

  3. How does NTK-Replay compare to ER with ρ\rho tuned on validation? How was ω\omega chosen?

  4. Which models are used in Tables 3–4?

5 comments

Dear Reviewer,

Thank you for your insightful comments and for highlighting several important aspects of our work. We have carefully considered each point and performed additional evaluations to address your concerns. Our responses are provided below.

Question 1: How does the predictor compare against squared-loss NTK and KMF1K_{MF}\mathbf{1}?

Response 1: To further evaluate our predictor, we compared it with the squared-loss NTK predictor and KMF1K_{MF}\mathbf{1}. As shown in Rebuttal Table 1, our method achieves the highest Pearson correlation and the lowest RMSE. Notably, although KMF1K_{MF}\mathbf{1} maintains relatively high correlations, its RMSE is substantially larger, indicating poor prediction accuracy in terms of response magnitude. These results highlight the advantage of our CE-based response formulation in capturing both the direction and magnitude of the observed training dynamics.

Rebuttal Table 1. Comparison of prediction performance among different predictors across three datasets. Each entry reports Pearson correlation / RMSE.

PredictorMMLU PearsonMMLU RMSEMedQA PearsonMedQA RMSECSQA PearsonCSQA RMSE
Ours0.7413.1400.8102.1740.6693.277
Squared loss0.4599.0370.6837.1630.66710.023
KMF1K_{MF}\mathbf{1}0.6896.594×1086.594 \times 10^{8}0.7906.874×1086.874 \times 10^{8}0.5783.441×1083.441 \times 10^{8}

Question 2: Do the correlations hold at standard learning rates or with Adam?

Response 2: To examine whether the observed correlations persist under different optimization settings, we conducted additional experiments using a 5× larger learning rate and AdamW. As shown in Rebuttal Table 2, the correlations remain consistently high across all three datasets under both settings, ranging from 0.898 to 0.954. These results demonstrate that the observed correlations are robust to changes in learning rate and optimizer choice, rather than being specific to the original SGD setting.

MethodMMLUMedQACSQA
5× learning rate0.9390.9530.927
AdamW0.9260.9540.898

Question 3: How does NTK-Replay compare to ER with tuned on validation? How was ω\omega chosen?

Response 3: We first examined how much computation is required to identify a suitable fixed replay weight for ER. Starting with ρ=1\rho=1, we doubled the replay weight at each search round, resulting in four configurations: ρ∈1,2,4,8\rho \in \\{1,2,4,8\\}. The cumulative wall-clock time is reported in Rebuttal Table 3 to account for the cost of all preceding search rounds.

The results show that identifying a competitive fixed replay weight requires substantial computational effort. After four search rounds, ER consumes 10.16, 12.48, and 3.32 hours on MMLU, MedQA, and CSQA, respectively. By comparison, NTK-Replay completes training in only 3.30, 4.04, and 1.31 hours, while achieving higher accuracy on all three datasets. This comparison highlights the advantage of adaptive replay-weight optimization over repeatedly training models with different fixed weights.

We empirically determined the default value of ω\omega. To examine sensitivity to ω\omega, we varied it over a 100-fold range as shown in Table 12. NTK-Replay maintains broadly comparable learning-forgetting trade-offs on MMLU, MedQA, and CSQA, with overall accuracy varying by only 0.24, 0.25, and 1.06 percentage points, respectively. In contrast, an 8-fold change in ER's replay weight produces larger accuracy variations of 3.96, 6.86, and 4.63 percentage points on the same three datasets, respectively. These results suggest that, the accuracy-forgetting trade-off is much less sensitive to ω\omega than to ER's replay weight, reducing the need for extensive hyperparameter tuning.

Taken together, these findings suggest that NTK-Replay offers a practical alternative to fixed-weight tuning, achieving a stronger acquisition-retention balance.

Rebuttal Table 3. Performance of fixed-weight ER search and NTK-Replay.

DatasetMethodAccuracy (%) ↑Learning (%) ↑Forgetting (%) ↓Total wall-clock time (h) ↓
MMLUER (1st search round, ρ=1\rho=1)54.8428.5226.092.54
MMLUER (2nd search round, ρ=2\rho=2)56.9926.8121.155.08
MMLUER (3rd search round, ρ=4\rho=4)57.8422.4316.517.62
MMLUER (4th search round, ρ=8\rho=8)58.8018.6112.0910.16
MMLUNTK-Replay60.3517.628.693.30
MedQAER (1st search round, ρ=1\rho=1)38.5822.1641.003.12
MedQAER (2nd search round, ρ=2\rho=2)41.9918.4428.736.24
MedQAER (3rd search round, ρ=4\rho=4)44.1812.8116.809.36
MedQAER (4th search round, ρ=8\rho=8)45.449.7910.2012.48
MedQANTK-Replay46.4612.4911.244.04
CSQAER (1st search round, ρ=1\rho=1)69.8124.6415.570.83
CSQAER (2nd search round, ρ=2\rho=2)71.2223.3313.291.66
CSQAER (3rd search round, ρ=4\rho=4)72.9620.7110.142.49
CSQAER (4th search round, ρ=8\rho=8)74.4418.377.423.32
CSQANTK-Replay77.2822.314.921.31

Question 4: Which models are used in Tables 3-4?

Response 4: Both Tables 3 and 4 report results using Qwen2.5-1.5B-Instruct. We will explicitly state the model name in both table captions to remove this ambiguity.

We are grateful for your constructive input, which has helped us better establish the strengths and limitations of our approach. We will carefully reflect these additional findings and clarifications in the revised manuscript. Thank you again for your consideration.

Reviewer_tzZPOpenReview ↗

Response to Authors

Thank authors' good response, you have address some of my concerns. Could the authors provide a fair comparison of training costs or clarify whether the cost of tuning for NTK-Replay is included?

Reviewer_a78QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper derives a local approximation to changes in correct-answer log odds, decomposing the response into transfer geometry, curvature and residual terms. It uses this view to connect projection, regularization and replay, and proposes NTK-Replay together with a factorized gradient sketch. The exposition has a consistent theme, but the connection between the derived stationary response and the training process used in the experiments remains insufficiently established.

Critical strengths
  1. The decomposition is interpretable. Retaining linearization remainders in the derivation helps distinguish the underlying expression from its practical approximation.

  2. The reported mean correlation of 0.796 across 32 model–dataset combinations is encouraging, with coverage across text, image-text and video-text settings.

  3. The paper states conditions for the connections between replay, regularization and projection rather than claiming unconditional equivalence.

  4. The appendix includes RMSE results, damping sensitivity and measured sketching costs. These provide useful evidence beyond correlation.

Critical weaknesses

The central issue is the correspondence between theory and optimization. Equations 15–18 use a stationary response for a damped local objective. Appendix B describes finite-horizon SGD without momentum or weight decay; it remains unclear how the centered penalty in Equation 2 is implemented and how damping relates to the optimization horizon. The remainder terms are identified, but their practical contribution is not quantified. Correlation alone also does not establish calibrated prediction of drift or actual forgetting. A strong first-order gradient-overlap baseline is needed to isolate the benefit of the curvature term. NTK-Replay improves acquisition but does not dominate retention: O-LoRA has substantially lower forgetting in Table 4. The practical comparison should therefore examine the acquisition–retention frontier under matched tuning and computation.

Questions for authors
  1. Is the centered damping penalty used during the reported SGD runs, and how is its coefficient related to training duration?

  2. What improvement remains over a scaled first-order gradient-overlap predictor, including sign prediction and identification of forgotten examples?

  3. Does NTK-Replay improve the acquisition–retention frontier when competing methods are tuned for several operating points?

2 comments

Dear Reviewer,

We sincerely appreciate your thoughtful comments and constructive suggestions. In response, we have conducted additional analyses and experiments to further validate our theoretical findings and evaluate the effectiveness and computational efficiency of NTK-Replay. Our point-by-point responses are provided below.

Question 1: Is the centered damping penalty used during the reported SGD runs, and how is its coefficient related to training duration?

Response 1: We apologize for the confusion caused by the unclear description in the original manuscript. Our theoretical derivation includes a damping term, whereas its coefficient is set to zero throughout the reported SGD training runs.

However, the NTK-based response matrix is not necessarily invertible. We therefore introduce damping when fitting the analytical predictor to ensure its invertibility. Table 7 shows that prediction correlations remain relatively stable across the evaluated damping values. We have clarified this distinction and specified the default damping value in Appendix D.4 of the revised manuscript.

Question 2: What improvement remains over a scaled first-order gradient-overlap predictor, including sign prediction and identification of forgotten examples?

Response 2: We compared the full second-order response with a scaled first-order gradient-overlap predictor using AUROC. For sign prediction, the positive label indicates that the observed drift is positive. For forgotten-example identification, a positive label indicates that an initially correct example becomes incorrect after training.

As shown in Rebuttal Table 1, the full second-order response improves AUROC from 92.57% to 94.61% for sign prediction and from 63.97% to 74.44% for forgotten-example identification. Notably, the substantial improvement in forgotten-example identification suggests that the second-order response provides additional predictive information beyond first-order gradient overlap, particularly in identifying examples that are ultimately forgotten.

We have added these results to Appendix F and included a reference in Section 4.2: “Appendix F compares alternative predictors and examines drift-sign prediction and forgotten-example identification.”

Rebuttal Table 1. Comparison of second-order and first-order predictors for drift sign prediction and forgotten-example identification (AUROC, %).

PredictorSign predictionForgotten-example identification
First-order92.5763.97
Second-order94.6174.44

Question 3: Does NTK-Replay improve the acquisition-retention frontier when competing methods are tuned for several operating points?

Response 3: To provide a more comprehensive comparison of the acquisition-retention trade-off, we evaluated competing methods under multiple hyperparameter configurations. Specifically, we varied OPGD's perturbation radius over {0.02, 0.05, 0.2}, EWC's regularization coefficient over {500, 5000, 50000}, DER++'s hyperparameters over {(0.5, 1), (1, 0.5), (0.75, 0.75)}, and O-LoRA's rank over {4, 8, 16, 32}.

As shown in Rebuttal Table 2, NTK-Replay achieves the highest accuracy across all three datasets. Although O-LoRA exhibits substantially lower forgetting, this comes at the cost of considerably less acquisition of new knowledge, resulting in lower overall accuracy according to Rebuttal Table 3. In contrast, NTK-Replay achieves a more favorable balance between knowledge acquisition and retention, learning substantially more new knowledge while keeping forgetting relatively low. These results demonstrate that NTK-Replay provides a better acquisition-retention trade-off across the evaluated baseline configurations.

We have added the baseline hyperparameter sensitivity results to Appendix I and updated Section 4.4 to state: “We further assess the sensitivity of baseline performance to hyperparameter choices in Appendix I.” We also emphasize in the revised manuscript that the primary advantage of NTK-Replay lies in achieving a better acquisition-retention trade-off, rather than merely reducing forgetting.

Rebuttal Table 2. Multiple hyperparameter configurations across different acquisition–retention operating points. Each entry reports Accuracy (%) / Forgetting (%).

MethodConfigurationMMLUMedQACSQA
OPGDb=0.02b=0.0249.16 / 36.7734.05 / 53.0767.12 / 20.99
OPGDb=0.05b=0.0548.78 / 37.6333.56 / 54.5067.12 / 21.04
OPGDb=0.20b=0.2048.20 / 39.0833.69 / 53.9067.46 / 20.45
EWCλ=500\lambda=50052.91 / 29.2836.08 / 47.3168.15 / 19.10
EWCλ=5000\lambda=500053.51 / 27.7537.10 / 44.6667.70 / 19.32
EWCλ=50000\lambda=5000053.52 / 26.1936.22 / 40.8368.11 / 18.33
DER++(α,β)=(0.5,1)(\alpha,\beta)=(0.5,1)56.27 / 21.5141.72 / 29.8371.13 / 14.05
DER++(α,β)=(1,0.5)(\alpha,\beta)=(1,0.5)56.12 / 20.6841.19 / 30.4371.33 / 13.33
DER++(α,β)=(0.75,0.75)(\alpha,\beta)=(0.75,0.75)56.34 / 20.6841.58 / 29.8371.09 / 13.78
O-LoRARank=457.69 / 2.6543.39 / 4.0175.47 / 1.26
O-LoRARank=857.50 / 1.4943.29 / 2.1275.88 / 0.59
O-LoRARank=1657.53 / 0.7443.09 / 1.4475.98 / 0.23
O-LoRARank=3257.57 / 0.2943.06 / 0.8375.95 / 0.14
NTK-Replay---60.35 / 8.6946.46 / 11.2477.28 / 4.92

Rebuttal Table 3. Comparison of knowledge acquisition and forgetting between O-LoRA and NTK-Replay across different adapter ranks. Each entry reports Accuracy (%) / Learning (%) / Forgetting (%).

MethodConfigurationMMLUMedQACSQA
O-LoRARank=457.69 / 4.03 / 2.6543.39 / 3.29 / 4.0175.47 / 1.99 / 1.26
O-LoRARank=857.50 / 2.01 / 1.4943.29 / 1.67 / 2.1275.88 / 1.56 / 0.59
O-LoRARank=1657.53 / 1.06 / 0.7443.09 / 0.81 / 1.4475.98 / 0.85 / 0.23
O-LoRARank=3257.57 / 0.56 / 0.2943.06 / 0.29 / 0.8375.95 / 0.43 / 0.14
NTK-Replay---60.35 / 17.62 / 8.6946.46 / 12.49 / 11.2477.28 / 22.31 / 4.92

We sincerely thank you for your time and valuable feedback. Your suggestions have helped us further strengthen the empirical evidence and improve the clarity of our manuscript. We will incorporate the corresponding clarifications and additional results into the revised version, and look forward to your further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.