- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates catastrophic forgetting during fine-tuning of pretrained models, focusing on how network connectivity affects the effectiveness of parameter freezing. The authors propose Guarded Freezing, which combines two strategies: (1) Removal-value, a capacity-based criterion for selecting important neurons when incoming trainable connections can be isolated, and (2) Drift-value, a forward-only metric for identifying weights whose updates may cause substantial output perturbations in Transformers. Experiments on VGG-8, six language models, and DINOv3 demonstrate improved retention of pretrained capabilities while maintaining comparable new-task performance. Drift-value generally outperforms random freezing and several importance-based baselines.
Critical strengths
- Meaningful Research Problem: The paper highlights an important limitation of parameter freezing: frozen weights may still produce altered outputs due to upstream updates. The connectivity-based analysis provides a valuable perspective on mitigating catastrophic forgetting.
- Computational Efficiency and Experimental Validation: The proposed Drift-value score requires only forward-pass calibration without gradient computation. Experiments across multiple language and vision models demonstrate improved knowledge retention over random freezing while maintaining comparable new-task performance.
Critical weaknesses
1.Limited Methodological Novelty: Removal-value produces the same within-layer ranking as the existing Capacity metric. Additionally, Drift-value shows no clear advantage over the simpler input-moment-only score, raising concerns about the necessity of the proposed formulations.
2.Insufficient Improvement over Strong Baselines: Drift-value does not demonstrate a clear advantage over Fisher across the six language models at 40 epochs. Its improvement over static Fisher at 160 epochs is limited to Qwen2.5-1.5B, weakening the evidence for its overall effectiveness.
3.Limited Real-World Validation: The language experiments primarily rely on memorizing 150 synthetic facts, while the main vision results use a high-learning-rate stress protocol. The effectiveness of the proposed method on realistic instruction fine-tuning and domain adaptation tasks remains unclear.
Questions for authors
- Given that Removal-value induces the same within-layer ranking as Capacity, and Appendix Table 12 shows no resolved improvement of Drift-value over the input-moment-only score, what additional empirical evidence supports the necessity of these formulations beyond simpler alternatives?
- Since Drift-value does not consistently outperform empirical Fisher at 40 epochs, could the authors quantify its actual calibration-time and memory advantages under matched conditions, and clarify when these savings justify the observed retention differences?
- Could the authors provide evidence that Drift-value remains effective under standard learning rates and realistic instruction fine-tuning tasks, rather than relying primarily on synthetic fact memorization and high-learning-rate stress tests?
4 comments
Additional results: instruction tuning, one epoch
Retention (minimum accuracy over commonsense, reasoning and knowledge)
| Model, rate | Pretrained | Full | Attention frozen | Random | Drift-value | Fisher | SSU |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B, 1e-4 | 0.540 | 0.498 ± 0.007 | 0.521 ± 0.007 | 0.560 ± 0.007 | 0.539 ± 0.007 | 0.533 ± 0.005 | 0.556 ± 0.005 |
| Qwen2.5-1.5B, 1e-4 | 0.640 | 0.510 ± 0.014 | 0.550 ± 0.014 | 0.580 ± 0.010 | 0.612 ± 0.016 | 0.607 ± 0.006 | 0.603 ± 0.006 |
| Qwen3-0.6B, 1e-4 | 0.450 | 0.377 ± 0.012 | 0.397 ± 0.008 | 0.430 ± 0.008 | 0.438 ± 0.009 | 0.443 ± 0.015 | 0.443 ± 0.010 |
| SmolLM2-1.7B, 1e-4 | 0.650 | 0.658 ± 0.004 | 0.673 ± 0.014 | 0.678 ± 0.003 | 0.663 ± 0.003 | 0.665 ± 0.000 | 0.673 ± 0.003 |
| SmolLM2-360M, 1e-4 | 0.535 | 0.510 ± 0.005 | 0.515 ± 0.008 | 0.513 ± 0.007 | 0.517 ± 0.006 | 0.513 ± 0.007 | 0.513 ± 0.006 |
| Qwen3-1.7B, 2e-5 | 0.540 | 0.596 ± 0.004 | 0.593 ± 0.003 | 0.589 ± 0.004 | 0.562 ± 0.004 | 0.553 ± 0.003 | 0.569 ± 0.005 |
| Qwen2.5-1.5B, 2e-5 | 0.640 | 0.632 ± 0.003 | 0.642 ± 0.007 | 0.630 ± 0.003 | 0.635 ± 0.003 | 0.630 ± 0.003 | 0.632 ± 0.002 |
| Qwen3-0.6B, 2e-5 | 0.450 | 0.453 ± 0.002 | 0.470 ± 0.003 | 0.467 ± 0.009 | 0.467 ± 0.007 | 0.475 ± 0.005 | 0.478 ± 0.009 |
| SmolLM2-1.7B, 2e-5 | 0.650 | 0.668 ± 0.002 | 0.678 ± 0.003 | 0.677 ± 0.003 | 0.667 ± 0.003 | 0.665 ± 0.000 | 0.667 ± 0.003 |
| SmolLM2-360M, 2e-5 | 0.535 | 0.528 ± 0.003 | 0.535 ± 0.005 | 0.538 ± 0.003 | 0.535 ± 0.003 | 0.538 ± 0.002 | 0.537 ± 0.002 |
Instruction quality (macro ROUGE-L on 119 unseen tasks)
| Model, rate | Pretrained | Full | Attention frozen | Random | Drift-value | Fisher | SSU |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B, 1e-4 | 0.052 | 0.394 ± 0.004 | 0.440 ± 0.004 | 0.467 ± 0.004 | 0.475 ± 0.004 | 0.481 ± 0.004 | 0.480 ± 0.006 |
| Qwen2.5-1.5B, 1e-4 | 0.088 | 0.365 ± 0.005 | 0.381 ± 0.007 | 0.425 ± 0.016 | 0.442 ± 0.008 | 0.446 ± 0.001 | 0.444 ± 0.000 |
| Qwen3-0.6B, 1e-4 | 0.052 | 0.351 ± 0.004 | 0.394 ± 0.005 | 0.425 ± 0.002 | 0.434 ± 0.003 | 0.438 ± 0.004 | 0.435 ± 0.007 |
| SmolLM2-1.7B, 1e-4 | 0.063 | 0.429 ± 0.004 | 0.440 ± 0.007 | 0.443 ± 0.009 | 0.445 ± 0.009 | 0.449 ± 0.010 | 0.450 ± 0.002 |
| SmolLM2-360M, 1e-4 | 0.056 | 0.376 ± 0.005 | 0.386 ± 0.008 | 0.396 ± 0.005 | 0.397 ± 0.002 | 0.393 ± 0.004 | 0.396 ± 0.004 |
| Qwen3-1.7B, 2e-5 | 0.052 | 0.507 ± 0.003 | 0.519 ± 0.003 | 0.523 ± 0.002 | 0.524 ± 0.002 | 0.523 ± 0.002 | 0.522 ± 0.001 |
| Qwen2.5-1.5B, 2e-5 | 0.088 | 0.494 ± 0.010 | 0.509 ± 0.006 | 0.519 ± 0.007 | 0.523 ± 0.010 | 0.523 ± 0.009 | 0.521 ± 0.012 |
| Qwen3-0.6B, 2e-5 | 0.052 | 0.450 ± 0.002 | 0.472 ± 0.008 | 0.477 ± 0.004 | 0.478 ± 0.002 | 0.479 ± 0.001 | 0.478 ± 0.007 |
| SmolLM2-1.7B, 2e-5 | 0.063 | 0.479 ± 0.006 | 0.472 ± 0.005 | 0.465 ± 0.002 | 0.465 ± 0.002 | 0.464 ± 0.001 | 0.465 ± 0.001 |
| SmolLM2-360M, 2e-5 | 0.056 | 0.402 ± 0.002 | 0.401 ± 0.005 | 0.398 ± 0.004 | 0.392 ± 0.004 | 0.393 ± 0.005 | 0.396 ± 0.005 |
Mean ± SE over three seeds. Pretrained is one evaluation. Every policy except full tuning freezes attention. Random, drift-value, Fisher and SSU also freeze 35.2% of feed-forward entries.
Additional results: instruction tuning, three epochs
Retention after three epochs (minimum accuracy over commonsense, reasoning and knowledge)
| Model, rate | Pretrained | Full | Attention frozen | Random | Drift-value | Fisher | SSU | Wanda | RIA |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B, 1e-4 | 0.540 | 0.447 ± 0.011 | 0.508 ± 0.016 | 0.518 ± 0.007 | 0.550 ± 0.010 | 0.532 ± 0.004 | 0.547 ± 0.016 | 0.542 ± 0.012 | 0.532 ± 0.011 |
| Qwen2.5-1.5B, 1e-4 | 0.640 | 0.433 ± 0.006 | 0.500 ± 0.008 | 0.558 ± 0.007 | 0.575 ± 0.009 | 0.572 ± 0.009 | 0.572 ± 0.002 | 0.562 ± 0.002 | 0.545 ± 0.012 |
| Qwen3-0.6B, 1e-4 | 0.450 | 0.338 ± 0.026 | 0.402 ± 0.024 | 0.418 ± 0.008 | 0.445 ± 0.006 | 0.435 ± 0.010 | 0.427 ± 0.003 | 0.442 ± 0.006 | 0.438 ± 0.009 |
| SmolLM2-1.7B, 1e-4 | 0.650 | 0.675 ± 0.008 | 0.672 ± 0.017 | 0.670 ± 0.006 | 0.670 ± 0.006 | 0.672 ± 0.006 | 0.682 ± 0.003 | 0.677 ± 0.003 | 0.677 ± 0.008 |
| SmolLM2-360M, 1e-4 | 0.535 | 0.533 ± 0.007 | 0.532 ± 0.003 | 0.527 ± 0.003 | 0.533 ± 0.002 | 0.528 ± 0.003 | 0.538 ± 0.002 | 0.533 ± 0.002 | 0.527 ± 0.004 |
| Qwen3-1.7B, 2e-5 | 0.540 | 0.597 ± 0.010 | 0.590 ± 0.003 | 0.605 ± 0.008 | 0.565 ± 0.003 | 0.563 ± 0.003 | 0.585 ± 0.009 | 0.597 ± 0.004 | 0.597 ± 0.004 |
| Qwen2.5-1.5B, 2e-5 | 0.640 | 0.620 ± 0.003 | 0.627 ± 0.002 | 0.642 ± 0.003 | 0.633 ± 0.009 | 0.627 ± 0.004 | 0.632 ± 0.008 | 0.640 ± 0.010 | 0.650 ± 0.005 |
| Qwen3-0.6B, 2e-5 | 0.450 | 0.463 ± 0.009 | 0.472 ± 0.008 | 0.477 ± 0.009 | 0.465 ± 0.003 | 0.467 ± 0.002 | 0.455 ± 0.003 | 0.463 ± 0.004 | 0.458 ± 0.007 |
| SmolLM2-1.7B, 2e-5 | 0.650 | 0.697 ± 0.004 | 0.695 ± 0.003 | 0.685 ± 0.003 | 0.663 ± 0.004 | 0.663 ± 0.003 | 0.670 ± 0.006 | 0.672 ± 0.002 | 0.677 ± 0.002 |
| SmolLM2-360M, 2e-5 | 0.535 | 0.527 ± 0.002 | 0.530 ± 0.003 | 0.530 ± 0.006 | 0.532 ± 0.006 | 0.533 ± 0.004 | 0.532 ± 0.006 | 0.533 ± 0.004 | 0.528 ± 0.007 |
Instruction quality after three epochs (macro ROUGE-L on 119 unseen tasks)
| Model, rate | Pretrained | Full | Attention frozen | Random | Drift-value | Fisher | SSU | Wanda | RIA |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B, 1e-4 | 0.052 | 0.349 ± 0.008 | 0.406 ± 0.003 | 0.444 ± 0.005 | 0.465 ± 0.003 | 0.467 ± 0.003 | 0.448 ± 0.010 | 0.443 ± 0.003 | 0.453 ± 0.005 |
| Qwen2.5-1.5B, 1e-4 | 0.088 | 0.313 ± 0.007 | 0.354 ± 0.008 | 0.403 ± 0.010 | 0.408 ± 0.002 | 0.411 ± 0.003 | 0.411 ± 0.003 | 0.411 ± 0.002 | 0.401 ± 0.005 |
| Qwen3-0.6B, 1e-4 | 0.052 | 0.300 ± 0.007 | 0.361 ± 0.001 | 0.404 ± 0.005 | 0.424 ± 0.006 | 0.430 ± 0.006 | 0.410 ± 0.004 | 0.405 ± 0.002 | 0.398 ± 0.001 |
| SmolLM2-1.7B, 1e-4 | 0.063 | 0.438 ± 0.006 | 0.467 ± 0.001 | 0.486 ± 0.004 | 0.483 ± 0.004 | 0.482 ± 0.001 | 0.485 ± 0.003 | 0.477 ± 0.001 | 0.482 ± 0.010 |
| SmolLM2-360M, 1e-4 | 0.056 | 0.399 ± 0.001 | 0.412 ± 0.002 | 0.412 ± 0.008 | 0.410 ± 0.004 | 0.403 ± 0.008 | 0.413 ± 0.010 | 0.413 ± 0.006 | 0.414 ± 0.006 |
| Qwen3-1.7B, 2e-5 | 0.052 | 0.512 ± 0.009 | 0.520 ± 0.003 | 0.525 ± 0.004 | 0.533 ± 0.004 | 0.528 ± 0.002 | 0.530 ± 0.004 | 0.533 ± 0.008 | 0.526 ± 0.005 |
| Qwen2.5-1.5B, 2e-5 | 0.088 | 0.486 ± 0.006 | 0.501 ± 0.003 | 0.513 ± 0.008 | 0.518 ± 0.004 | 0.524 ± 0.008 | 0.517 ± 0.004 | 0.519 ± 0.003 | 0.519 ± 0.000 |
| Qwen3-0.6B, 2e-5 | 0.052 | 0.444 ± 0.004 | 0.467 ± 0.006 | 0.468 ± 0.001 | 0.478 ± 0.005 | 0.485 ± 0.002 | 0.477 ± 0.005 | 0.478 ± 0.003 | 0.473 ± 0.001 |
| SmolLM2-1.7B, 2e-5 | 0.063 | 0.488 ± 0.003 | 0.488 ± 0.008 | 0.481 ± 0.004 | 0.481 ± 0.001 | 0.479 ± 0.002 | 0.484 ± 0.004 | 0.486 ± 0.003 | 0.483 ± 0.003 |
| SmolLM2-360M, 2e-5 | 0.056 | 0.418 ± 0.004 | 0.407 ± 0.005 | 0.400 ± 0.001 | 0.388 ± 0.001 | 0.393 ± 0.003 | 0.389 ± 0.002 | 0.400 ± 0.003 | 0.401 ± 0.003 |
Mean ± SE over three seeds. Three epochs of the same instruction data, rates and evaluation, with the learning-rate schedule spanning all three epochs. Pretrained is one evaluation. Every policy except full tuning freezes attention. Random, drift-value, Fisher, SSU, Wanda and RIA also freeze 35.2% of feed-forward entries. SSU rounds to whole columns.
Revised Manuscript
Thank you for your time and your criticism, we have revised the manuscript based on your feedback.
Our contribution is a principle for choosing what to freeze and a method built on it, Guarded Freezing. Freezing alone cannot preserve a network's performance while trainable parameters can still influence the frozen core, so the choice depends on connectivity. With the paths into the frozen core kept, no tested rule on VGG-8 separates from a random core (Figure 2a). With the paths cut, removal-value, a neuron's share of its layer's capacity, places the core across layers and retains 6 to 12 points more than DEFT at 40 to 70% frozen, while still learning the new task (Figure 2a, Table 9). With the paths open, an update disturbs a layer's output, and drift-value prices that disturbance from one forward pass. In contrast, Empirical Fisher, the strongest competitor in this regime, needs gradients. After 40 epochs drift-value stays within 1.7 points of it on six language models, and on Qwen3-1.7B its calibration pass is 2.6 times faster at half the memory.
The revised manuscript marks every new or changed passage, table and figure in blue. Its main changes are:
- Where the paths can be cut, complete grids compare the selection rules at every share from 40 to 90% of neurons frozen on three CNNs (previously one), and a second transfer pair is added (Section 4, Tables 9 to 12). On VGG-8 from 50 to 70%, removal-value keeps the most old-task accuracy and reaches the new-task target in all 20 transfers. At 70%, the rules within the decision threshold of it reach that target in 2 of 20. On VGG-11 and VanillaNet-6 it retains the most at every share. Section 3.1 states removal-value as the share form of HOPE's removal cost, and Appendix A gives its exact properties.
- Where the paths stay open, complete grids add MAS, Model-Dowser, S2FT and EWC on Qwen3-1.7B, and source and target Fisher on DINOv3 (Sections 5 and 6, Tables 13 and 14). On Qwen3-1.7B no rule retains more than drift-value beyond the decision threshold at any share, and empirical Fisher, which stays within it, needs a backward pass and twice the memory (Table 17). New tables freeze one block at a time and replace the invented facts with instruction tuning on Super-NaturalInstructions (Tables 15, 16, 18 and 19).
- Clarity. The abstract states the question and the method, the introduction ends with a list of contributions, every term is defined at first use, and the method is placed in the parameter-isolation family of continual learning (Sections 1 and 2).
We would be happy to answer any additional questions that would help you revisit your assessment.
Thank you for your time and effort reviewing our submission. You highlight that frozen weights may still produce altered outputs because of upstream updates. That is the core of our paper, and the evidence you requested strengthens both of its scores. Where the frozen neurons can be isolated, our removal-value sets a new state of the art it keeps 5.22 ± 0.51 points more than the best method at equal accuracy (Fig. 2, L193-197), and in the new comparison 11 to 17 points more than raw capacity. Where the neurons cannot be isolated, as in transformers, drift-value matches Fisher, the strongest published score, at half the GPU memory, with no gradients, on the fact task and on realistic instruction tuning.
W1/Q1 (new evidence): Removal-value produces the same within-layer ranking as the existing Capacity metric. [...] what additional empirical evidence supports the necessity? Within a layer the rankings agree, as Section 3.1 states (L140-141). The contribution is the allocation across layers, and we show it matters. On VGG-8 adapted from CIFAR-100 to SVHN (the setting of Figure 2, 20 paired scenarios), with the same number of frozen units and incoming paths cut, removal-value retains 11.4 ± 1.1 and 16.5 ± 1.6 points more CIFAR-100 accuracy than raw capacity at 40% and 55% frozen, and 1.7 ± 0.7 more at 70%. As shown before, it also beats DEFT by 5.22 ± 0.51 points (L196). Where the core can be isolated, we set a new state of the art. You are right that drift-value provides an alternative, it is competitive, without the need of gradients of Fisher (the best performing method where isolation is not possible).
W1/W2: Drift-value shows no clear advantage over the simpler input-moment-only score [...] over Fisher We provide this ablation already in appendix Table 12: the input moment carries most of the effect. What matters is dropping weight magnitude. Wanda and RIA keep it and trail by 0.044 ± 0.005 and 0.073 ± 0.009 retention on Qwen3-1.7B (Table 1, L264). Against Fisher, drift-value reaches competitive retention on all six models (Fig. 3) without any gradient. Over long training, Fisher must be recomputed every 20 epochs, with backward passes each time, to reach what one drift-value mask computed once already holds (0.441 vs 0.443 on Qwen2.5-1.5B after 160 epochs, L1329-1331).
Q2 (new evidence): [...] quantify its actual calibration-time and memory advantages under matched conditions Drift-value calibration peaks at 7.12 GiB against Fisher's 14.37 GiB and runs 2.6 times faster (0.371 s vs 0.970 s) on Qwen3-1.7B with the same 12 prompts and GPU. Retention is equal (Fig. 3). This is decisive as the GPU memory limits the model that can be calibrated.
W3/Q3 (new evidence): [...] effective under standard learning rates and realistic instruction fine-tuning tasks Yes. For realistic instruction tuning we ran five language models tuned on Super-NaturalInstructions and scored by ROUGE-L on 119 unseen tasks, with retention measured as in the paper (worst accuracy over HellaSwag, ARC-Easy and SciQ). Freezing attention and the 35.2% of feed-forward entries with the highest drift-value beats full tuning on both axes. On Qwen3-1.7B, Qwen2.5-1.5B and Qwen3-0.6B at learning rate 1e-4, it keeps 0.04 to 0.10 more retention and 0.08 more ROUGE-L after one epoch, and 0.10 to 0.14 more retention and 0.10 to 0.12 more ROUGE-L after three epochs. In every model and rate, at both durations, it stays within 0.02 retention and 0.01 ROUGE-L of Fisher, without the need for gradients. At the standard learning rate 2e-5, after one epoch, forgetting is small for every method, and drift-value still improves ROUGE-L over full tuning on all three Qwen models with retention at or above pretrained on four of five models. Full tables follow in a separate comment. We already show realistic domain adaptation with the DINOv3 experiment (Section 6): the model moves from images to point clouds, and drift-value keeps 0.440 image accuracy where full fine-tuning keeps 0.025 (pretrained 0.728, Appendix Table 6). Its high learning rate is a stress test by design, where forgetting is largest and selection matters most.
We hope this helps you revisit your assessment, and we welcome further questions.