- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses catastrophic forgetting in LLM fine-tuning and proposes FINCH, which adjusts the learning rate based on the current training loss without modifying the training objective. The authors prove that the per-step forgetting is upper-bounded by a term proportional to the learning rate times the square root of the mini-batch loss. Experiments on three task types (knowledge acquisition, scientific reasoning, and low-resource language adaptation) across four models show that FINCH matches standard SFT on the new tasks while reducing net degradation of general capabilities by about 94% on average.
Critical strengths
Forgetting caused by fine-tuning is a common problem in real-world deployment. The method only changes the learning rate, leaves the loss function untouched, adds almost no computational overhead, and is easy to plug into existing training pipelines.
The experiments cover four models from three families (4B–32B, including both dense and MoE architectures) and three task types, and compare against nine existing methods.
Critical weaknesses
The key step in Theorem 1 is that the gradient norm is bounded by C × , which motivates scaling the learning rate by 1/. However, all experiments use AdamW. Adam normalizes gradients by their second-moment estimates, so the actual step size for each parameter is mainly determined by the learning rate, and this step of the derivation no longer holds under Adam. The authors should provide an analysis for Adam.
The theory defines forgetting as the increase in old-task log loss (Definition 1), but the experiments never measure L_old itself. Instead, they report multiple-choice accuracy and generation-based metrics such as IFEval. The relationship between the two is neither justified theoretically nor verified empirically.
η_max is fixed at 5e-5 across all experiments, but the authors neither explain how this value was chosen nor provide a sensitivity analysis.
Experiments are conducted on only three tasks, and the knowledge acquisition task uses synthetic data (TOFU). This is not enough to show that the method works in real specialized domains; additional data from domains such as medicine or law is recommended. In addition, the win-tie rate used for language adaptation is already close to the 99% ceiling on some models, which limits its ability to distinguish between methods.
Among the four benchmarks used to measure retention of general capabilities, HellaSwag, WinoGrande, and MMLU are relatively old, mostly multiple-choice, and fairly easy, with many models already near saturation. Newer and harder benchmarks, such as code generation and long-context tasks, should be included.
Questions for authors
See above
4 comments
W5. Harder benchmarks for measuring retention
We add LiveCodeBench v6 [2] for code generation and RULER at 64K [3] for long-context evaluation. We measure changes from the pretrained model after fine-tuning on each of the six adaptation tasks (Appendix B.10, Tables 22 and 23). FINCH reduces the mean drop relative to SFT on both benchmarks and both model sizes.
Table 2. Mean score changes from the pretrained model across the six adaptation tasks, in percentage points. Detailed results for all baselines and tasks are provided in Appendix B.10, Tables 22 and 23.
LiveCodeBench v6 (Δ percentage points)
| Model | SFT | FINCH |
|---|---|---|
| Qwen3-4B-Instruct-2507 | −14.0 | −1.3 |
| Qwen3-32B | −9.7 | −1.8 |
RULER 64K (Δ percentage points)
| Model | SFT | FINCH |
|---|---|---|
| Qwen3-4B-Instruct-2507 | −5.1 | −1.2 |
| Qwen3-32B | −6.5 | −0.9 |
We also report retention on TruthfulQA and HaluEval (Section 5 and Appendix B.9).
References
[1] Zheng, Lucia, et al. "When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings." Proceedings of the eighteenth international conference on artificial intelligence and law. 2021.
[2] Jain, Naman, et al. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.” International Conference on Learning Representations (2025).
[3] Hsieh, Cheng-Ping, et al. "RULER: What's the real context size of your long-context language models?." arXiv preprint arXiv:2404.06654 (2024).
Please let us know if you have further concerns, and please consider raising the score if we have cleared existing concerns – thank you so much!
Looking Forward to Further Discussion
Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.
As the discussion period is ending in three days, we look forward to your thoughts on our response. Let us know if there is anything more we can do to address your comments!
Thanks again for your time and constructive feedback!
Dear Reviewer,
Please carefully review the authors' rebuttal and respond to each of your concerns. If any concerns remain unresolved, clearly specify what is missing and request concrete clarifications or additional evidence.
Bests,
Your AC.
We thank the reviewer for the feedback and thoughtful suggestions. We appreciate the reviewer’s recognition of FINCH’s practical simplicity and the breadth of our experiments.
We address each concern below. We have also updated the submitted PDF with additional results, clarifications, and edits.
W1. Applicability of the theoretical analysis to AdamW
We agree that Theorem 1 is specific to SGD. As stated in Section 3.3 and footnote 1, our analysis motivates FINCH’s learning-rate schedule but does not prove it is optimal. We validate FINCH in practice with AdamW on three tasks across four models from three families and on three additional adaptation tasks using the two models. We add a discussion of which steps of the analysis carry over to AdamW and where its update rule changes the argument (Appendix A.4).
Further, as an empirical test, on Qwen3-4B-Instruct-2507 knowledge acquisition, we measure the change in old-task negative log-likelihood (NLL) before and after a single AdamW update. Its Pearson correlation with the square root of the current minibatch loss is r = 0.72 (Appendix A.4). This suggests support for FINCH’s premise that higher-loss steps are associated with larger increases in old-task loss. We identify extending the theoretical guarantee to AdamW as future work (Section 7).
W2. Measurement of old-task loss
We benchmark accuracy for old-task performance since that is the standard way for evaluating LLMs in practice and in other forgetting mitigation papers.
To test whether we also maintain old-task loss, we measure changes from the pretrained model after fine-tuning on the KA task. We score the correct answer given each question and average the NLL and accuracy changes across MMLU, HellaSwag, and WinoGrande (Appendix B.11, Table 24, and Table 3 of the paper).
FINCH preserves both benchmark accuracy and old-task NLL better than SFT on both models.
W3. Choice and sensitivity of the learning-rate ceiling
We use ηmax=5×10−5 to cap the learning rate when the smoothed training loss becomes small (Section 3.3). We test ceilings of 5×10−5, 10−4, and 2×10−4 (Appendix B.7.7). At the best base learning rate for each task, target-task performance and retention are identical across the three settings because the adaptive learning rate stays below 5×10−5 throughout training.
W4. Additional evaluation on a specialized domain
We add CaseHOLD [1], a legal reasoning task on Qwen3-4B-Instruct-2507 and Qwen3-32B (Appendix B.8.3, Table 18). FINCH matches or slightly improves on SFT’s CaseHOLD accuracy and significantly improves old-task retention on both models. We also add two additional tasks: APIGen-MT and Verilog generation, as detailed in our response to Reviewer k6ku (W1).
Table 1. CaseHOLD accuracy (%). Avg. Δ is the mean change from the pretrained model across the four old-task benchmarks, in percentage points. Detailed results are provided in Appendix B.8.3, Table 18.