acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

Abstract

Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as catastrophic forgetting. Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences, but these tokens are essential for learning new tasks, especially those with poor pretraining coverage. In such settings, hard tokens should still contribute to learning, so forgetting must be controlled without suppressing them. We identify a simple mechanism for doing so: per-step forgetting is bounded by the product of the learning rate and the square root of the current training loss. This suggests that high-loss batches are especially prone to inducing forgetting. Motivated by this observation, we introduce FINCH, a loss-adaptive learning-rate schedule that reduces the learning rate on high-loss batches and increases it as the model converges, while leaving the fine-tuning objective unchanged. Across knowledge acquisition, science, and low-resource language adaptation benchmarks on four models from three families, FINCH reduces forgetting by 94% on average while matching the task performance of standard fine-tuning. On Qwen3-4B knowledge acquisition, FINCH cuts TruthfulQA degradation by and reverses HaluEval degradation. Overall, our results show that learning-rate schedules are an effective tool to shape model behavior during fine-tuning, beyond just target-task optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Catastrophic Forgetting, Fine-tuning, LLMs, continual learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GV5SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses catastrophic forgetting in LLM fine-tuning and proposes FINCH, which adjusts the learning rate based on the current training loss without modifying the training objective. The authors prove that the per-step forgetting is upper-bounded by a term proportional to the learning rate times the square root of the mini-batch loss. Experiments on three task types (knowledge acquisition, scientific reasoning, and low-resource language adaptation) across four models show that FINCH matches standard SFT on the new tasks while reducing net degradation of general capabilities by about 94% on average.

Critical strengths

Forgetting caused by fine-tuning is a common problem in real-world deployment. The method only changes the learning rate, leaves the loss function untouched, adds almost no computational overhead, and is easy to plug into existing training pipelines.

The experiments cover four models from three families (4B–32B, including both dense and MoE architectures) and three task types, and compare against nine existing methods.

Critical weaknesses

The key step in Theorem 1 is that the gradient norm is bounded by C × L\sqrt{\mathcal{L}}, which motivates scaling the learning rate by 1/L\sqrt{\mathcal{L}}. However, all experiments use AdamW. Adam normalizes gradients by their second-moment estimates, so the actual step size for each parameter is mainly determined by the learning rate, and this step of the derivation no longer holds under Adam. The authors should provide an analysis for Adam.

The theory defines forgetting as the increase in old-task log loss (Definition 1), but the experiments never measure L_old itself. Instead, they report multiple-choice accuracy and generation-based metrics such as IFEval. The relationship between the two is neither justified theoretically nor verified empirically.

η_max is fixed at 5e-5 across all experiments, but the authors neither explain how this value was chosen nor provide a sensitivity analysis.

Experiments are conducted on only three tasks, and the knowledge acquisition task uses synthetic data (TOFU). This is not enough to show that the method works in real specialized domains; additional data from domains such as medicine or law is recommended. In addition, the win-tie rate used for language adaptation is already close to the 99% ceiling on some models, which limits its ability to distinguish between methods.

Among the four benchmarks used to measure retention of general capabilities, HellaSwag, WinoGrande, and MMLU are relatively old, mostly multiple-choice, and fairly easy, with many models already near saturation. Newer and harder benchmarks, such as code generation and long-context tasks, should be included.

Questions for authors

See above

4 comments

We thank the reviewer for the feedback and thoughtful suggestions. We appreciate the reviewer’s recognition of FINCH’s practical simplicity and the breadth of our experiments.

We address each concern below. We have also updated the submitted PDF with additional results, clarifications, and edits.

W1. Applicability of the theoretical analysis to AdamW

We agree that Theorem 1 is specific to SGD. As stated in Section 3.3 and footnote 1, our analysis motivates FINCH’s learning-rate schedule but does not prove it is optimal. We validate FINCH in practice with AdamW on three tasks across four models from three families and on three additional adaptation tasks using the two models. We add a discussion of which steps of the analysis carry over to AdamW and where its update rule changes the argument (Appendix A.4).

Further, as an empirical test, on Qwen3-4B-Instruct-2507 knowledge acquisition, we measure the change in old-task negative log-likelihood (NLL) before and after a single AdamW update. Its Pearson correlation with the square root of the current minibatch loss is r = 0.72 (Appendix A.4). This suggests support for FINCH’s premise that higher-loss steps are associated with larger increases in old-task loss. We identify extending the theoretical guarantee to AdamW as future work (Section 7).

W2. Measurement of old-task loss

We benchmark accuracy for old-task performance since that is the standard way for evaluating LLMs in practice and in other forgetting mitigation papers.

To test whether we also maintain old-task loss, we measure changes from the pretrained model after fine-tuning on the KA task. We score the correct answer given each question and average the NLL and accuracy changes across MMLU, HellaSwag, and WinoGrande (Appendix B.11, Table 24, and Table 3 of the paper).

ModelSFT Avg. Δ\Delta NLLFINCH Avg. Δ\Delta NLLSFT Avg. Δ\Delta accuracy (pp)FINCH Avg. Δ\Delta accuracy (pp)
Qwen3-4B-Instruct-2507+3.96+3.96−1.29-1.29−9.6-9.60.00.0
Qwen3-32B+6.98+6.98−0.28-0.28−8.6-8.6−1.1-1.1

FINCH preserves both benchmark accuracy and old-task NLL better than SFT on both models.

W3. Choice and sensitivity of the learning-rate ceiling

We use ηmax⁡=5×10−5\eta_{\max}=5\times10^{-5} to cap the learning rate when the smoothed training loss becomes small (Section 3.3). We test ceilings of 5×10−55\times10^{-5}, 10−410^{-4}, and 2×10−42\times10^{-4} (Appendix B.7.7). At the best base learning rate for each task, target-task performance and retention are identical across the three settings because the adaptive learning rate stays below 5×10−55\times10^{-5} throughout training.

W4. Additional evaluation on a specialized domain

We add CaseHOLD [1], a legal reasoning task on Qwen3-4B-Instruct-2507 and Qwen3-32B (Appendix B.8.3, Table 18). FINCH matches or slightly improves on SFT’s CaseHOLD accuracy and significantly improves old-task retention on both models. We also add two additional tasks: APIGen-MT and Verilog generation, as detailed in our response to Reviewer k6ku (W1).

Table 1. CaseHOLD accuracy (%). Avg. Δ\Delta is the mean change from the pretrained model across the four old-task benchmarks, in percentage points. Detailed results are provided in Appendix B.8.3, Table 18.

MethodQwen3-4B-Instruct-2507 accuracyQwen3-4B-Instruct-2507 Avg. Δ\DeltaQwen3-32B accuracyQwen3-32B Avg. Δ\Delta
Base67.00.071.00.0
SFT77.0−9.882.0−9.3
SFT (small LR)69.0−0.775.0−1.6
FLOW74.5−10.970.5−8.6
DFT68.5−0.373.0+0.2
TALR71.0−0.473.00.0
STM68.50.068.5−1.2
L2 Reg75.0−14.180.0−9.7
WiSE-FT73.5−6.679.5−10.4
LoRA79.0−11.282.0−5.2
FINCH77.5−1.982.0+0.6

W5. Harder benchmarks for measuring retention

We add LiveCodeBench v6 [2] for code generation and RULER at 64K [3] for long-context evaluation. We measure changes from the pretrained model after fine-tuning on each of the six adaptation tasks (Appendix B.10, Tables 22 and 23). FINCH reduces the mean drop relative to SFT on both benchmarks and both model sizes.

Table 2. Mean score changes from the pretrained model across the six adaptation tasks, in percentage points. Detailed results for all baselines and tasks are provided in Appendix B.10, Tables 22 and 23.

LiveCodeBench v6 (Δ percentage points)

ModelSFTFINCH
Qwen3-4B-Instruct-2507−14.0−1.3
Qwen3-32B−9.7−1.8

RULER 64K (Δ percentage points)

ModelSFTFINCH
Qwen3-4B-Instruct-2507−5.1−1.2
Qwen3-32B−6.5−0.9

We also report retention on TruthfulQA and HaluEval (Section 5 and Appendix B.9).

References

[1] Zheng, Lucia, et al. "When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings." Proceedings of the eighteenth international conference on artificial intelligence and law. 2021.

[2] Jain, Naman, et al. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.” International Conference on Learning Representations (2025).

[3] Hsieh, Cheng-Ping, et al. "RULER: What's the real context size of your long-context language models?." arXiv preprint arXiv:2404.06654 (2024).

Please let us know if you have further concerns, and please consider raising the score if we have cleared existing concerns – thank you so much!

AuthorsOpenReview ↗

Looking Forward to Further Discussion

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

As the discussion period is ending in three days, we look forward to your thoughts on our response. Let us know if there is anything more we can do to address your comments!

Thanks again for your time and constructive feedback!

Dear Reviewer,

Please carefully review the authors' rebuttal and respond to each of your concerns. If any concerns remain unresolved, clearly specify what is missing and request concrete clarifications or additional evidence.

Bests,

Your AC.

Reviewer_k6kuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FINCH, an adaptive learning rate scheduler that uses a smaller learning rate during the stage when the model is most prone to forgetting to alleviate forgetting, and increases the learning rate when the model loss is relatively low to strengthen learning of downstream capabilities.

Critical strengths

This paper transforms catastrophic forgetting into a learning rate scheduling problem by approximating single-step forgetting as the product of the learning rate and the square root of the batch loss. The experiments also demonstrate that FINCH achieves a good tradeoff between downstream performance and forgetting.

Critical weaknesses

I think the paper's experiments do not fully prove the effectiveness of FINCH. Because a learning rate scheduling approach that is almost opposite to normal model training can easily fail to learn properly on many long-horizon or difficult tasks, I hope the authors can supplement performance results on some Agent, high-difficulty math, and coding tasks.

Questions for authors

If we simply reverse the currently common learning rate scheduling approach (where the learning rate first increases to the maximum and then gradually decreases) into the opposite form (where the learning rate gradually increases and finally rises to lr max), would this approach be a stronger baseline?

4 comments

We thank the reviewer for the feedback and thoughtful suggestions. We appreciate the reviewer’s recognition of FINCH’s theoretical motivation and the breadth of our experiments.

We address each concern below. We have also updated the submitted PDF with additional results, clarifications, and edits.

W1. Additional results on agent, math, and coding tasks

We add an agentic task (APIGen-MT [1]) and a coding task (Verilog generation using RTLCoder [2]) on Qwen3-4B-Instruct-2507 and Qwen3-32B (Section 4.1 and Appendix B.8, Tables 16 and 17 of the paper).

Table 1. Qwen3-4B-Instruct-2507 Target scores are percentages. Avg. Δ is the mean change from the pretrained model across the four old-task benchmarks, in percentage points.

MethodAPIGen-MT targetAPIGen-MT Avg. ΔVerilog targetVerilog Avg. Δ
Base75.10.024.00.0
SFT83.8−9.629.0−7.5
SFT (small LR)75.9−1.825.5−1.1
FLOW83.5−9.725.5−6.2
DFT77.4+0.723.5+0.2
TALR79.6+0.423.5+0.8
STM75.5+0.222.5+0.8
L2 Reg83.8−9.229.0−7.5
WiSE-FT83.1−9.230.5−8.5
LoRA82.9−3.627.0−12.2
FINCH85.2+0.730.0+0.9

Table 2. Qwen3-32B

MethodAPIGen-MT targetAPIGen-MT Avg. ΔVerilog targetVerilog Avg. Δ
Base74.90.015.00.0
SFT84.2−10.126.5−8.5
SFT (small LR)79.8−0.719.0−2.8
FLOW83.6−9.924.0−9.2
DFT78.6+0.117.5+1.3
TALR78.6+1.218.0+0.6
STM74.1+0.516.0+1.8
L2 Reg83.1−10.025.0−8.9
WiSE-FT84.9−3.425.5−5.4
LoRA83.9−5.026.0−17.0
FINCH86.1−0.325.0−1.3

FINCH improves retention over SFT in all four agentic and coding settings. It improves tool-call accuracy on both model sizes and Verilog pass@1 on Qwen3-4B-Instruct-2507. On Qwen3-32B Verilog, FINCH’s pass@1 is 1.5 points lower than SFT’s, and its Avg. Δ is 7.2 points higher.

We also fine-tune Llama-3-8B-Instruct on OpenR1-Math-220k and evaluate on MATH-500 (Section 6 and Appendix B.7.1, Table 8). FINCH achieves 33.3% mean accuracy, close to SFT (33.6%) and FLOW (33.8%). Its Avg. Δ is −0.1, compared with −4.8 for SFT. FLOW and DFT also preserve old-task performance (−0.4 and +0.4 Avg. Δ, respectively). This is consistent with our premise that existing forgetting mitigation methods mitigate forgetting on tasks with low number of rare tokens. For example, in this setting, the pretrained model assigns log probability below −5 to 3.1% of MATH-500 tokens, compared with 23.8% of KA tokens. On harder adaptation tasks such as KA and LA, these baselines often incur substantial forgetting or achieve lower target-task performance.

Q1. A steadily increasing learning-rate baseline

We add a linear ramp from zero to the peak learning rate over the full training run (Section 6 and Appendix B.7.4, Table 12). Under the same training budget on Qwen3-4B-Instruct-2507 KA, the ramp achieves 81.8% accuracy and −8.7 Avg. Δ, compared with 83.3% accuracy and +0.1 Avg. Δ for FINCH. FINCH improves over the ramp on both target accuracy and retention. We report comparisons with other learning-rate schedules in our response to Reviewer 1eRr (W3).

References

[1] Prabhakar, Akshara, et al. “APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay.” Advances in Neural Information Processing Systems 2025

[2] Liu, Shang, et al. “RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation Technique.” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 2024

Please let us know if you have further concerns, and please consider raising the score if we have cleared existing concerns – thank you so much!

AuthorsOpenReview ↗

Looking Forward to Further Discussion

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

As the discussion period is ending in three days, we look forward to your thoughts on our response. Let us know if there is anything more we can do to address your comments!

Thanks again for your time and constructive feedback!

Thank you for your reply, I think it addresses my concerns. I will keep my score.

Thank you for your reply and positive assessment! We are glad our rebuttal has addressed your concerns.

Reviewer_1eRrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FINCH, a loss-adaptive learning-rate scheduling method designed to reduce catastrophic forgetting during LLM fine-tuning. The authors show theoretically that per-step forgetting is bounded by a quantity related to the learning rate and the square root of the training loss. Based on this observation, FINCH uses smaller learning rates when the training loss is high and gradually increases them as the loss decreases, without modifying the fine-tuning objective.

Critical strengths

The paper presents an interesting approach to reducing catastrophic forgetting by adjusting the learning rate rather than modifying the training loss or suppressing high-loss tokens. This allows the model to learn difficult examples while better preserving its existing capabilities.

The paper provides a theoretical motivation for its method by deriving a bound connecting the learning rate, training loss, and forgetting. This helps explain why smaller learning rates on high-loss batches could reduce forgetting and provides a clear motivation for the proposed schedule.

The paper includes extensive experiments across different model families, model sizes, and adaptation tasks. The comparisons with several existing methods show that FINCH achieves a better balance between learning new tasks and preserving previous capabilities.

Critical weaknesses

The theoretical analysis derives the forgetting bound under SGD updates, whereas the actual experiments use AdamW. Since AdamW has different update dynamics, it is unclear whether the theoretical guarantees directly apply to the optimizer used in practice. The paper could provide additional analysis connecting the theoretical results to AdamW.

Most experiments focus on adapting a pretrained model to a single new task, with only one sequential experiment involving three tasks. Since the method aims to reduce catastrophic forgetting, evaluating it over longer sequences of tasks would help establish whether its benefits persist under repeated updates.

FINCH increases the learning rate as training loss decreases, but the paper does not fully establish whether the improvements come specifically from adapting to the loss or from using a more favorable learning-rate schedule. Although the authors compare against standard warmup and smaller learning rates, a more direct comparison with alternative increasing or adaptive schedules under matched training budgets would help isolate the contribution of loss-based adaptation.

Questions for authors

Please refer to the weaknesses listed above.

4 comments

We thank the reviewer for the feedback and thoughtful suggestions. We appreciate the reviewer’s recognition of FINCH’s theoretical motivation and the breadth of our experiments.

We address each concern below. We have also updated the submitted PDF with additional results, clarifications, and edits.

W1. Additional analysis connecting the theoretical results to AdamW

Thank you for the suggestion. We add an analysis of how our theory applies to AdamW (Appendix A.4). Two parts of the analysis carry over to AdamW: (1) the bound on the minibatch gradient norm and (2) the Taylor bound that relates forgetting to the size of the parameter update. The step that connects these two bounds relies on the SGD update rule. Under AdamW, the update also depends on momentum, second-moment normalization, and weight decay. Therefore, the bound does not directly carry over to AdamW.

We also empirically test the relationship between minibatch loss and forgetting on old tasks. On Qwen3-4B knowledge acquisition (KA), we measure old-task negative log-likelihood before and after a single AdamW update at selected training steps (Appendix A.4). We find a Pearson correlation of r = 0.72 between the square root of the current minibatch loss and the one-step change in old-task loss.

Our theoretical analysis uses SGD to motivate FINCH’s learning-rate schedule. Our extensive experiments (that all use AdamW) validate FINCH in practice. In the conclusion (Section 7), we identify the scope of the theoretical guarantee as a limitation and extending it to AdamW as a direction for future work.

W2. Longer sequential fine-tuning

We extend the sequential evaluation from three tasks to a ten-task sequence from Continual Reasoning Gym [1] (Section 4.2). Table 1 shows that SFT and FINCH reach similar mean accuracy immediately after learning each task. At the end of the sequence, SFT’s mean accuracy drops by 4.7 percentage points, while FINCH’s drops by only 0.6 points. Appendix B.6.1 and Table 7 of the paper give the per-task results and further details.

Table 1. Mean accuracy (%) across the ten tasks. Base is measured before any fine-tuning. After task is measured on each task immediately after fine-tuning on it. Final is measured on each task after completing the full sequence.

MethodBaseAfter taskFinal
SFT15.039.234.5
FINCH15.039.438.8

W3. Comparison with other increasing or adaptive schedules

We compare with five additional learning-rate schedules on Qwen3-4B KA (Section 6 and Appendix B.7.4, Table 12). Table 2 summarizes the results for a full-run linear ramp, linear warmup followed by inverse-square-root decay [2], SGDR [3], GreedyLR [4], and Sensitivity-Aware Warm-Up (SAWU) [5]. The linear ramp is an increasing schedule, SGDR is cyclical, and GreedyLR and SAWU are adaptive.

Table 2. Learning-rate schedule comparison on Qwen3-4B KA. Avg. Δ is the mean change from the pretrained model across the four old-task benchmarks, in percentage points.

ScheduleKA accuracy (%)Avg. Δ
SFT, 5% warmup + cosine decay83.5−9.6
Linear ramp to peak LR81.8−8.7
Linear warmup + inverse-square-root decay [2]82.0−8.3
SGDR [3]80.0−7.1
GreedyLR [4]80.3−11.5
SAWU [5]81.3−11.2
FINCH83.3+0.1

FINCH achieves higher KA accuracy and better retention than all five alternatives in this setting. It comes within 0.2 percentage points of the default SFT schedule on KA accuracy and improves Avg. Δ by 9.7 points.

References

[1] Luo, Lirui, et al. "Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR." arXiv preprint arXiv:2608.18574 (2026).

[2] Vaswani, Ashish, et al. "Attention Is All You Need." Advances in Neural Information Processing Systems (2017).

[3] Loshchilov, Ilya, and Frank Hutter. "SGDR: Stochastic Gradient Descent with Warm Restarts." International Conference on Learning Representations (2017).

[4] Subramanian, Shreyas, Bala Krishnamoorthy, and Pranav Murthy. "Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence." arXiv preprint arXiv:2512.14527 (2025).

[5] Chen, Yile, et al. "A Better Start: Sensitivity-Aware Warm-Up for Robust and Efficient Fine-Tuning." Proceedings of the AAAI Conference on Artificial Intelligence (2026).

Please let us know if you have further concerns, and please consider raising the score if we have cleared existing concerns – thank you so much!

AuthorsOpenReview ↗

Looking Forward to Futher Discussion

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

As the discussion period is ending in three days, we look forward to your thoughts on our response. Let us know if there is anything more we can do to address your comments!

Thanks again for your time and constructive feedback!

Thank you to the authors for the rebuttal. Most of my concerns have been addressed, so I will maintain my positive rating.

Thank you for your reply and positive assessment! We are glad our rebuttal has addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.