- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper applies a second-order local expansion to cross-entropy and a first-order expansion to model outputs. It decomposes drift on the mastered set into transfer geometry, curvature response, and residual signal, then derives effective kernels for projection, quadratic regularization, and replay. NTK-Replay selects the replay weight by minimizing normalized predicted drift plus a penalty on replay strength. Experiments span text, image, and speech models, reporting drift correlations for 32 model–dataset combinations and replay results on three text datasets.
Critical strengths
- The unified formulation is somewhat novel. The framework explicitly incorporates cross-entropy curvature and describes projection, regularization, and replay through a common effective-kernel language. For the three mitigation strategies, reported drift correlations are approximately 0.92–0.96.
- The empirical scope and cost analysis are substantial. Drift prediction is evaluated across three modalities, five datasets, and twelve models. The appendices also report RMSE, damping sensitivity, sketching time, and memory use.
Critical weaknesses
- The theoretical predictor does not match the training dynamics. Equations (15)–(20) describe the stationary point of a damped local surrogate, while the experiments use ten epochs of SGD without weight decay; the damping coefficient is set only from the kernel spectral radius. High Pearson correlation does not establish scale accuracy, and some RMSE values exceed 3 log-odds units.
- The source of NTK-Replay's improvement is not isolated. The evaluation lacks a thoroughly tuned fixed replay weight, an oracle weight selected using observed validation drift, and random or grid-selected weight controls. The reported time-to-quality comparison also does not replace a comparison at equal total compute.
Questions for authors
- If the predictor is replaced by finite-step, undamped linearized SGD dynamics matching the experiments, what are the resulting correlation, RMSE, calibration slope, and intercept? Please also report Jacobian drift and update norms during training, and relate them to prediction error.
- With the buffer, total compute, and tuning budget held fixed, does NTK-Replay still outperform a well-tuned fixed-weight ER baseline, an oracle replay weight, and a simple grid search? Please report new-task learning, old-task retention, the selected replay weights, and total wall-clock time separately.
3 comments
Question 2: With the buffer, total compute, and tuning budget held fixed, does NTK-Replay still outperform a well-tuned fixed-weight ER baseline, an oracle replay weight, and a simple grid search? Please report new-task learning, old-task retention, the selected replay weights, and total wall-clock time separately.
Response 2: To evaluate the computational cost of tuning fixed replay weights, we conducted a progressive grid search for ER, starting from a replay weight of 1 and doubling it at each successive search round (). We accumulated the wall-clock time across successive search rounds to reflect the actual tuning cost.
As shown in Rebuttal Table 4, after four search rounds, the cumulative wall-clock time for ER reaches 10.16 hours on MMLU, 12.48 hours on MedQA, and 3.32 hours on CSQA, compared with only 3.30, 4.04, and 1.31 hours for NTK-Replay, respectively. Given the high computational cost of LLM training, extensive hyperparameter searches are particularly time-consuming. Beyond this computational advantage, NTK-Replay also achieves higher accuracy across all three datasets.
We further conducted a hyperparameter search for DER++-based replay using three different ((\alpha,\beta)) configurations. As shown in Rebuttal Table 5, these configurations also require considerable cumulative training time, yet none matches the overall performance of NTK-Replay.
Overall, these results demonstrate that NTK-Replay can adaptively determine replay weights to achieve a more favorable acquisition-retention trade-off, while avoiding the substantial computational cost associated with repeatedly training models under different fixed-weight configurations.
The ER and DER++ search results are now reported in Appendix J. Section 4.4 states: “Appendix J investigates replay performance and cumulative tuning cost for ER and DER++.”
Rebuttal Table 4. Comparison of progressive fixed-weight ER search and NTK-Replay in terms of knowledge acquisition, forgetting, and cumulative computational cost. Each entry reports Accuracy (%) / Learning (%) / Forgetting (%) / Total wall-clock time (h).
| Dataset | Method | Accuracy (%) ↑ | Learning (%) ↑ | Forgetting (%) ↓ | Total wall-clock time (h) ↓ |
|---|---|---|---|---|---|
| MMLU | ER (1st round, ) | 54.84 | 28.52 | 26.09 | 2.54 |
| MMLU | ER (2nd round, ) | 56.99 | 26.81 | 21.15 | 5.08 |
| MMLU | ER (3rd round, ) | 57.84 | 22.43 | 16.51 | 7.62 |
| MMLU | ER (4th round, ) | 58.80 | 18.61 | 12.09 | 10.16 |
| MMLU | NTK-Replay | 60.35 | 17.62 | 8.69 | 3.30 |
| MedQA | ER (1st round, ) | 38.58 | 22.16 | 41.00 | 3.12 |
| MedQA | ER (2nd round, ) | 41.99 | 18.44 | 28.73 | 6.24 |
| MedQA | ER (3rd round, ) | 44.18 | 12.81 | 16.80 | 9.36 |
| MedQA | ER (4th round, ) | 45.44 | 9.79 | 10.20 | 12.48 |
| MedQA | NTK-Replay | 46.46 | 12.49 | 11.24 | 4.04 |
| CSQA | ER (1st round, ) | 69.81 | 24.64 | 15.57 | 0.83 |
| CSQA | ER (2nd round, ) | 71.22 | 23.33 | 13.29 | 1.66 |
| CSQA | ER (3rd round, ) | 72.96 | 20.71 | 10.14 | 2.49 |
| CSQA | ER (4th round, ) | 74.44 | 18.37 | 7.42 | 3.32 |
| CSQA | NTK-Replay | 77.28 | 22.31 | 4.92 | 1.31 |
Rebuttal Table 5. Progressive hyperparameter search for DER++ and comparison with NTK-Replay. Each entry reports Accuracy (%) / Learning (%) / Forgetting (%) / Total wall-clock time (h).
| Dataset | Method | Accuracy (%) ↑ | Learning (%) ↑ | Forgetting (%) ↓ | Total wall-clock time (h) ↓ |
|---|---|---|---|---|---|
| MMLU | DER++ (1st round, ) | 56.27 | 26.19 | 21.51 | 3.75 |
| MMLU | DER++ (2nd round, ) | 56.12 | 24.73 | 20.68 | 10.90 |
| MMLU | DER++ (3rd round, ) | 56.34 | 25.24 | 20.68 | 18.21 |
| MMLU | NTK-Replay | 60.35 | 17.62 | 8.69 | 3.30 |
| MedQA | DER++ (1st round, ) | 41.72 | 20.02 | 29.83 | 4.26 |
| MedQA | DER++ (2nd round, ) | 41.19 | 19.56 | 30.43 | 12.31 |
| MedQA | DER++ (3rd round, ) | 41.58 | 19.79 | 29.83 | 19.20 |
| MedQA | NTK-Replay | 46.46 | 12.49 | 11.24 | 4.04 |
| CSQA | DER++ (1st round, ) | 71.13 | 24.32 | 14.05 | 1.44 |
| CSQA | DER++ (2nd round, ) | 71.33 | 22.90 | 13.33 | 2.79 |
| CSQA | DER++ (3rd round, ) | 71.09 | 23.33 | 13.78 | 4.16 |
| CSQA | NTK-Replay | 77.28 | 22.31 | 4.92 | 1.31 |
Question 3: High Pearson correlation does not establish scale accuracy, and some RMSE values exceed 3 log-odds units.
Response 3: We acknowledge that the RMSE exceeds 3 log-odds on some datasets. To further evaluate the scale accuracy of our predictor, we compared our full CE response with the NTK baseline from Liu et al. (TPAMI 2026). As shown in Rebuttal Table 6, our method achieves substantially lower RMSE across all three datasets, demonstrating considerably better prediction accuracy than the existing NTK-based approach.
Rebuttal Table 6. RMSE comparison between our full CE response and the NTK baseline from TPAMI 2026.
| Predictor | MMLU | MedQA | CSQA |
|---|---|---|---|
| Full CE response | 3.140 | 2.174 | 3.277 |
| TPAMI 2026 | 9.037 | 7.163 | 10.023 |
[1] J Liu et al. Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective. TPAMI, 2026.
We sincerely thank you for your time and valuable feedback. Your comments have encouraged us to further investigate the theoretical and empirical aspects of our approach. We will incorporate the additional analyses, experimental results, and corresponding clarifications into the revised manuscript. We look forward to your further feedback.
Dear Reviewer,
We sincerely appreciate your thoughtful review and constructive suggestions. In response to your comments, we have conducted additional theoretical analyses and experiments, as detailed below.
Question 1: If the predictor is replaced by finite-step, undamped linearized SGD dynamics matching the experiments, what are the resulting correlation, RMSE, calibration slope, and intercept? Please also report Jacobian drift and update norms during training, and relate them to prediction error.
Response 1: To investigate the finite-step, undamped linearized SGD dynamics, we compared predictions obtained by recursively evolving the dynamics up to the 10th epoch with those derived from the stationary-point solution. As shown in Rebuttal Table 1, the two predictions yield nearly identical Pearson correlations, RMSEs, calibration slopes, and intercepts.
To understand this similarity, we further examined the cross-entropy loss at the end of each epoch in Rebuttal Table 2. The loss decreases progressively and approaches a plateau by the 10th epoch, suggesting that the models are already close to convergence. Consequently, the finite-step dynamics at the 10th epoch produce predictions that are nearly identical to those obtained from the stationary-point solution.
We have added the finite-step prediction results to Appendix H and updated Section 4.2 to state: “We study the effect of a finite prediction horizon in Appendix H.”
Rebuttal Table 1. Steady-state versus 10-epoch finite-step predictions.
Rebuttal Table 2. End-of-epoch training cross-entropy loss.
We additionally tracked the Jacobian drift and parameter update norms throughout training and report them alongside the prediction RMSE in Rebuttal Table 3. Both Jacobian drift and parameter update norms consistently increase with training epochs. Meanwhile, the prediction RMSE also increases progressively. Notably, the growth in Jacobian drift closely accompanies the increase in prediction error, suggesting that the evolving Jacobian contributes to the degradation of prediction accuracy under the fixed-Jacobian approximation. The increasing parameter update norms further indicate that the model progressively moves away from its initial parameter state, providing a plausible explanation for the observed Jacobian evolution.
Rebuttal Table 3. Epoch-wise RMSE, Jacobian drift, and update norms.