acceptodds
Under review as a conference paper at ICLR 2027

What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Models

Abstract

When adapting pre-trained models through fine-tuning, freezing weights alone might not preserve performance, as updates elsewhere can change the inputs to the frozen core, ultimately affecting overall performance. We first analyse the case where a selected frozen core can be isolated and propose _removal-value_, a capacity-based score that approximates HOPE's removal cost averaged over removal orders. We show that in VGG-8, cutting paths from trainable neurons into a frozen core makes selection using this score useful: frozen preserves percentage points more old-task accuracy than DEFT at similar new-task accuracy. In transformers, shared residual streams leave paths into frozen neurons open. For this case, we derive _drift-value_, a forward-only proxy for the output disturbance from updating each weight entry under a local update model. In language models, at 40 epochs, this policy exceeds adapted Wanda and RIA freezing scores in settings with substantial retention loss, while its differences from Fisher remain unresolved. After 160 epochs on Qwen2.5-1.5B, it retains more than static Fisher. In DINOv3 vision-transformer adaptation to point clouds, drift-value retains image accuracy versus for a random mask of the same count. These results motivate _Guarded Freezing_: select by removal-value when incoming paths are cut, and by drift-value when they remain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
selective fine-tuning, parameter freezing, knowledge retention, catastrophic forgetting, neural network connectivity, drift sensitivity, model pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gi6ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates catastrophic forgetting during fine-tuning of pretrained models, focusing on how network connectivity affects the effectiveness of parameter freezing. The authors propose Guarded Freezing, which combines two strategies: (1) Removal-value, a capacity-based criterion for selecting important neurons when incoming trainable connections can be isolated, and (2) Drift-value, a forward-only metric for identifying weights whose updates may cause substantial output perturbations in Transformers. Experiments on VGG-8, six language models, and DINOv3 demonstrate improved retention of pretrained capabilities while maintaining comparable new-task performance. Drift-value generally outperforms random freezing and several importance-based baselines.

Critical strengths
  1. Meaningful Research Problem: The paper highlights an important limitation of parameter freezing: frozen weights may still produce altered outputs due to upstream updates. The connectivity-based analysis provides a valuable perspective on mitigating catastrophic forgetting.
  2. Computational Efficiency and Experimental Validation: The proposed Drift-value score requires only forward-pass calibration without gradient computation. Experiments across multiple language and vision models demonstrate improved knowledge retention over random freezing while maintaining comparable new-task performance.
Critical weaknesses

1.Limited Methodological Novelty: Removal-value produces the same within-layer ranking as the existing Capacity metric. Additionally, Drift-value shows no clear advantage over the simpler input-moment-only score, raising concerns about the necessity of the proposed formulations.

2.Insufficient Improvement over Strong Baselines: Drift-value does not demonstrate a clear advantage over Fisher across the six language models at 40 epochs. Its improvement over static Fisher at 160 epochs is limited to Qwen2.5-1.5B, weakening the evidence for its overall effectiveness.

3.Limited Real-World Validation: The language experiments primarily rely on memorizing 150 synthetic facts, while the main vision results use a high-learning-rate stress protocol. The effectiveness of the proposed method on realistic instruction fine-tuning and domain adaptation tasks remains unclear.

Questions for authors
  1. Given that Removal-value induces the same within-layer ranking as Capacity, and Appendix Table 12 shows no resolved improvement of Drift-value over the input-moment-only score, what additional empirical evidence supports the necessity of these formulations beyond simpler alternatives?
  2. Since Drift-value does not consistently outperform empirical Fisher at 40 epochs, could the authors quantify its actual calibration-time and memory advantages under matched conditions, and clarify when these savings justify the observed retention differences?
  3. Could the authors provide evidence that Drift-value remains effective under standard learning rates and realistic instruction fine-tuning tasks, rather than relying primarily on synthetic fact memorization and high-learning-rate stress tests?
4 comments

Thank you for your time and effort reviewing our submission. You highlight that frozen weights may still produce altered outputs because of upstream updates. That is the core of our paper, and the evidence you requested strengthens both of its scores. Where the frozen neurons can be isolated, our removal-value sets a new state of the art it keeps 5.22 ± 0.51 points more than the best method at equal accuracy (Fig. 2, L193-197), and in the new comparison 11 to 17 points more than raw capacity. Where the neurons cannot be isolated, as in transformers, drift-value matches Fisher, the strongest published score, at half the GPU memory, with no gradients, on the fact task and on realistic instruction tuning.

W1/Q1 (new evidence): Removal-value produces the same within-layer ranking as the existing Capacity metric. [...] what additional empirical evidence supports the necessity? Within a layer the rankings agree, as Section 3.1 states (L140-141). The contribution is the allocation across layers, and we show it matters. On VGG-8 adapted from CIFAR-100 to SVHN (the setting of Figure 2, 20 paired scenarios), with the same number of frozen units and incoming paths cut, removal-value retains 11.4 ± 1.1 and 16.5 ± 1.6 points more CIFAR-100 accuracy than raw capacity at 40% and 55% frozen, and 1.7 ± 0.7 more at 70%. As shown before, it also beats DEFT by 5.22 ± 0.51 points (L196). Where the core can be isolated, we set a new state of the art. You are right that drift-value provides an alternative, it is competitive, without the need of gradients of Fisher (the best performing method where isolation is not possible).

W1/W2: Drift-value shows no clear advantage over the simpler input-moment-only score [...] over Fisher We provide this ablation already in appendix Table 12: the input moment carries most of the effect. What matters is dropping weight magnitude. Wanda and RIA keep it and trail by 0.044 ± 0.005 and 0.073 ± 0.009 retention on Qwen3-1.7B (Table 1, L264). Against Fisher, drift-value reaches competitive retention on all six models (Fig. 3) without any gradient. Over long training, Fisher must be recomputed every 20 epochs, with backward passes each time, to reach what one drift-value mask computed once already holds (0.441 vs 0.443 on Qwen2.5-1.5B after 160 epochs, L1329-1331).

Q2 (new evidence): [...] quantify its actual calibration-time and memory advantages under matched conditions Drift-value calibration peaks at 7.12 GiB against Fisher's 14.37 GiB and runs 2.6 times faster (0.371 s vs 0.970 s) on Qwen3-1.7B with the same 12 prompts and GPU. Retention is equal (Fig. 3). This is decisive as the GPU memory limits the model that can be calibrated.

W3/Q3 (new evidence): [...] effective under standard learning rates and realistic instruction fine-tuning tasks Yes. For realistic instruction tuning we ran five language models tuned on Super-NaturalInstructions and scored by ROUGE-L on 119 unseen tasks, with retention measured as in the paper (worst accuracy over HellaSwag, ARC-Easy and SciQ). Freezing attention and the 35.2% of feed-forward entries with the highest drift-value beats full tuning on both axes. On Qwen3-1.7B, Qwen2.5-1.5B and Qwen3-0.6B at learning rate 1e-4, it keeps 0.04 to 0.10 more retention and 0.08 more ROUGE-L after one epoch, and 0.10 to 0.14 more retention and 0.10 to 0.12 more ROUGE-L after three epochs. In every model and rate, at both durations, it stays within 0.02 retention and 0.01 ROUGE-L of Fisher, without the need for gradients. At the standard learning rate 2e-5, after one epoch, forgetting is small for every method, and drift-value still improves ROUGE-L over full tuning on all three Qwen models with retention at or above pretrained on four of five models. Full tables follow in a separate comment. We already show realistic domain adaptation with the DINOv3 experiment (Section 6): the model moves from images to point clouds, and drift-value keeps 0.440 image accuracy where full fine-tuning keeps 0.025 (pretrained 0.728, Appendix Table 6). Its high learning rate is a stress test by design, where forgetting is largest and selection matters most.

We hope this helps you revisit your assessment, and we welcome further questions.

Additional results: instruction tuning, one epoch

Retention (minimum accuracy over commonsense, reasoning and knowledge)

Model, ratePretrainedFullAttention frozenRandomDrift-valueFisherSSU
Qwen3-1.7B, 1e-40.5400.498 ± 0.0070.521 ± 0.0070.560 ± 0.0070.539 ± 0.0070.533 ± 0.0050.556 ± 0.005
Qwen2.5-1.5B, 1e-40.6400.510 ± 0.0140.550 ± 0.0140.580 ± 0.0100.612 ± 0.0160.607 ± 0.0060.603 ± 0.006
Qwen3-0.6B, 1e-40.4500.377 ± 0.0120.397 ± 0.0080.430 ± 0.0080.438 ± 0.0090.443 ± 0.0150.443 ± 0.010
SmolLM2-1.7B, 1e-40.6500.658 ± 0.0040.673 ± 0.0140.678 ± 0.0030.663 ± 0.0030.665 ± 0.0000.673 ± 0.003
SmolLM2-360M, 1e-40.5350.510 ± 0.0050.515 ± 0.0080.513 ± 0.0070.517 ± 0.0060.513 ± 0.0070.513 ± 0.006
Qwen3-1.7B, 2e-50.5400.596 ± 0.0040.593 ± 0.0030.589 ± 0.0040.562 ± 0.0040.553 ± 0.0030.569 ± 0.005
Qwen2.5-1.5B, 2e-50.6400.632 ± 0.0030.642 ± 0.0070.630 ± 0.0030.635 ± 0.0030.630 ± 0.0030.632 ± 0.002
Qwen3-0.6B, 2e-50.4500.453 ± 0.0020.470 ± 0.0030.467 ± 0.0090.467 ± 0.0070.475 ± 0.0050.478 ± 0.009
SmolLM2-1.7B, 2e-50.6500.668 ± 0.0020.678 ± 0.0030.677 ± 0.0030.667 ± 0.0030.665 ± 0.0000.667 ± 0.003
SmolLM2-360M, 2e-50.5350.528 ± 0.0030.535 ± 0.0050.538 ± 0.0030.535 ± 0.0030.538 ± 0.0020.537 ± 0.002

Instruction quality (macro ROUGE-L on 119 unseen tasks)

Model, ratePretrainedFullAttention frozenRandomDrift-valueFisherSSU
Qwen3-1.7B, 1e-40.0520.394 ± 0.0040.440 ± 0.0040.467 ± 0.0040.475 ± 0.0040.481 ± 0.0040.480 ± 0.006
Qwen2.5-1.5B, 1e-40.0880.365 ± 0.0050.381 ± 0.0070.425 ± 0.0160.442 ± 0.0080.446 ± 0.0010.444 ± 0.000
Qwen3-0.6B, 1e-40.0520.351 ± 0.0040.394 ± 0.0050.425 ± 0.0020.434 ± 0.0030.438 ± 0.0040.435 ± 0.007
SmolLM2-1.7B, 1e-40.0630.429 ± 0.0040.440 ± 0.0070.443 ± 0.0090.445 ± 0.0090.449 ± 0.0100.450 ± 0.002
SmolLM2-360M, 1e-40.0560.376 ± 0.0050.386 ± 0.0080.396 ± 0.0050.397 ± 0.0020.393 ± 0.0040.396 ± 0.004
Qwen3-1.7B, 2e-50.0520.507 ± 0.0030.519 ± 0.0030.523 ± 0.0020.524 ± 0.0020.523 ± 0.0020.522 ± 0.001
Qwen2.5-1.5B, 2e-50.0880.494 ± 0.0100.509 ± 0.0060.519 ± 0.0070.523 ± 0.0100.523 ± 0.0090.521 ± 0.012
Qwen3-0.6B, 2e-50.0520.450 ± 0.0020.472 ± 0.0080.477 ± 0.0040.478 ± 0.0020.479 ± 0.0010.478 ± 0.007
SmolLM2-1.7B, 2e-50.0630.479 ± 0.0060.472 ± 0.0050.465 ± 0.0020.465 ± 0.0020.464 ± 0.0010.465 ± 0.001
SmolLM2-360M, 2e-50.0560.402 ± 0.0020.401 ± 0.0050.398 ± 0.0040.392 ± 0.0040.393 ± 0.0050.396 ± 0.005

Mean ± SE over three seeds. Pretrained is one evaluation. Every policy except full tuning freezes attention. Random, drift-value, Fisher and SSU also freeze 35.2% of feed-forward entries.

Additional results: instruction tuning, three epochs

Retention after three epochs (minimum accuracy over commonsense, reasoning and knowledge)

Model, ratePretrainedFullAttention frozenRandomDrift-valueFisherSSUWandaRIA
Qwen3-1.7B, 1e-40.5400.447 ± 0.0110.508 ± 0.0160.518 ± 0.0070.550 ± 0.0100.532 ± 0.0040.547 ± 0.0160.542 ± 0.0120.532 ± 0.011
Qwen2.5-1.5B, 1e-40.6400.433 ± 0.0060.500 ± 0.0080.558 ± 0.0070.575 ± 0.0090.572 ± 0.0090.572 ± 0.0020.562 ± 0.0020.545 ± 0.012
Qwen3-0.6B, 1e-40.4500.338 ± 0.0260.402 ± 0.0240.418 ± 0.0080.445 ± 0.0060.435 ± 0.0100.427 ± 0.0030.442 ± 0.0060.438 ± 0.009
SmolLM2-1.7B, 1e-40.6500.675 ± 0.0080.672 ± 0.0170.670 ± 0.0060.670 ± 0.0060.672 ± 0.0060.682 ± 0.0030.677 ± 0.0030.677 ± 0.008
SmolLM2-360M, 1e-40.5350.533 ± 0.0070.532 ± 0.0030.527 ± 0.0030.533 ± 0.0020.528 ± 0.0030.538 ± 0.0020.533 ± 0.0020.527 ± 0.004
Qwen3-1.7B, 2e-50.5400.597 ± 0.0100.590 ± 0.0030.605 ± 0.0080.565 ± 0.0030.563 ± 0.0030.585 ± 0.0090.597 ± 0.0040.597 ± 0.004
Qwen2.5-1.5B, 2e-50.6400.620 ± 0.0030.627 ± 0.0020.642 ± 0.0030.633 ± 0.0090.627 ± 0.0040.632 ± 0.0080.640 ± 0.0100.650 ± 0.005
Qwen3-0.6B, 2e-50.4500.463 ± 0.0090.472 ± 0.0080.477 ± 0.0090.465 ± 0.0030.467 ± 0.0020.455 ± 0.0030.463 ± 0.0040.458 ± 0.007
SmolLM2-1.7B, 2e-50.6500.697 ± 0.0040.695 ± 0.0030.685 ± 0.0030.663 ± 0.0040.663 ± 0.0030.670 ± 0.0060.672 ± 0.0020.677 ± 0.002
SmolLM2-360M, 2e-50.5350.527 ± 0.0020.530 ± 0.0030.530 ± 0.0060.532 ± 0.0060.533 ± 0.0040.532 ± 0.0060.533 ± 0.0040.528 ± 0.007

Instruction quality after three epochs (macro ROUGE-L on 119 unseen tasks)

Model, ratePretrainedFullAttention frozenRandomDrift-valueFisherSSUWandaRIA
Qwen3-1.7B, 1e-40.0520.349 ± 0.0080.406 ± 0.0030.444 ± 0.0050.465 ± 0.0030.467 ± 0.0030.448 ± 0.0100.443 ± 0.0030.453 ± 0.005
Qwen2.5-1.5B, 1e-40.0880.313 ± 0.0070.354 ± 0.0080.403 ± 0.0100.408 ± 0.0020.411 ± 0.0030.411 ± 0.0030.411 ± 0.0020.401 ± 0.005
Qwen3-0.6B, 1e-40.0520.300 ± 0.0070.361 ± 0.0010.404 ± 0.0050.424 ± 0.0060.430 ± 0.0060.410 ± 0.0040.405 ± 0.0020.398 ± 0.001
SmolLM2-1.7B, 1e-40.0630.438 ± 0.0060.467 ± 0.0010.486 ± 0.0040.483 ± 0.0040.482 ± 0.0010.485 ± 0.0030.477 ± 0.0010.482 ± 0.010
SmolLM2-360M, 1e-40.0560.399 ± 0.0010.412 ± 0.0020.412 ± 0.0080.410 ± 0.0040.403 ± 0.0080.413 ± 0.0100.413 ± 0.0060.414 ± 0.006
Qwen3-1.7B, 2e-50.0520.512 ± 0.0090.520 ± 0.0030.525 ± 0.0040.533 ± 0.0040.528 ± 0.0020.530 ± 0.0040.533 ± 0.0080.526 ± 0.005
Qwen2.5-1.5B, 2e-50.0880.486 ± 0.0060.501 ± 0.0030.513 ± 0.0080.518 ± 0.0040.524 ± 0.0080.517 ± 0.0040.519 ± 0.0030.519 ± 0.000
Qwen3-0.6B, 2e-50.0520.444 ± 0.0040.467 ± 0.0060.468 ± 0.0010.478 ± 0.0050.485 ± 0.0020.477 ± 0.0050.478 ± 0.0030.473 ± 0.001
SmolLM2-1.7B, 2e-50.0630.488 ± 0.0030.488 ± 0.0080.481 ± 0.0040.481 ± 0.0010.479 ± 0.0020.484 ± 0.0040.486 ± 0.0030.483 ± 0.003
SmolLM2-360M, 2e-50.0560.418 ± 0.0040.407 ± 0.0050.400 ± 0.0010.388 ± 0.0010.393 ± 0.0030.389 ± 0.0020.400 ± 0.0030.401 ± 0.003

Mean ± SE over three seeds. Three epochs of the same instruction data, rates and evaluation, with the learning-rate schedule spanning all three epochs. Pretrained is one evaluation. Every policy except full tuning freezes attention. Random, drift-value, Fisher, SSU, Wanda and RIA also freeze 35.2% of feed-forward entries. SSU rounds to whole columns.

AuthorsOpenReview ↗

Revised Manuscript

Thank you for your time and your criticism, we have revised the manuscript based on your feedback.

Our contribution is a principle for choosing what to freeze and a method built on it, Guarded Freezing. Freezing alone cannot preserve a network's performance while trainable parameters can still influence the frozen core, so the choice depends on connectivity. With the paths into the frozen core kept, no tested rule on VGG-8 separates from a random core (Figure 2a). With the paths cut, removal-value, a neuron's share of its layer's capacity, places the core across layers and retains 6 to 12 points more than DEFT at 40 to 70% frozen, while still learning the new task (Figure 2a, Table 9). With the paths open, an update disturbs a layer's output, and drift-value prices that disturbance from one forward pass. In contrast, Empirical Fisher, the strongest competitor in this regime, needs gradients. After 40 epochs drift-value stays within 1.7 points of it on six language models, and on Qwen3-1.7B its calibration pass is 2.6 times faster at half the memory.

The revised manuscript marks every new or changed passage, table and figure in blue. Its main changes are:

  • Where the paths can be cut, complete grids compare the selection rules at every share from 40 to 90% of neurons frozen on three CNNs (previously one), and a second transfer pair is added (Section 4, Tables 9 to 12). On VGG-8 from 50 to 70%, removal-value keeps the most old-task accuracy and reaches the new-task target in all 20 transfers. At 70%, the rules within the decision threshold of it reach that target in 2 of 20. On VGG-11 and VanillaNet-6 it retains the most at every share. Section 3.1 states removal-value as the share form of HOPE's removal cost, and Appendix A gives its exact properties.
  • Where the paths stay open, complete grids add MAS, Model-Dowser, S2FT and EWC on Qwen3-1.7B, and source and target Fisher on DINOv3 (Sections 5 and 6, Tables 13 and 14). On Qwen3-1.7B no rule retains more than drift-value beyond the decision threshold at any share, and empirical Fisher, which stays within it, needs a backward pass and twice the memory (Table 17). New tables freeze one block at a time and replace the invented facts with instruction tuning on Super-NaturalInstructions (Tables 15, 16, 18 and 19).
  • Clarity. The abstract states the question and the method, the introduction ends with a list of contributions, every term is defined at first use, and the method is placed in the parameter-isolation family of continual learning (Sections 1 and 2).

We would be happy to answer any additional questions that would help you revisit your assessment.

Reviewer_vvToOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Based on what I understand of this paper, the authors argue that freezing task specific parameters during fine-tuning in order to prevent catastrophic forgetting, does not protect the frozen computation if trainable parameters upstream can still change its inputs. The authors hence propose two selection scores (i) removal-value for cases where trainable-to-frozen connections are cut, and (ii) drift-value (a forward-only estimate of output disturbance from weight updates) for cases where they remain open, as in transformers. The experiments cover a VGG-8 transfer task, six language models learning synthetic facts, and DINOv3 adapted to point clouds. The paper tackles an interesting and relevant problem in finetuning, which is preventing catastrophic forgetting on the pretrained task while trying to maximize performance on the finetuning task. And some of the insights provided in the paper for maintaining the frozen backbone using drift and retention scores are valuable. However, in its current form the paper is nearly impossible to follow and would require a significant rewrite for the general audience to understand and make this a good submission.

Critical strengths
  1. The observation that frozen weights with drifting inputs are not really protected is a relevant one, and the VGG-8 isolated-vs-exposed experiment illustrates it well, but I do not understand why the authors chose this specific (relatively outdated) CNN model.
  2. The drift-value consistently beats random masks of equal size across models and only requires one forward pass.
Critical weaknesses
  1. Clarity is the main problem.

    • The abstract and introduction do not state the research question or situate it in the literature.
    • Methods central to the paper (HOPE, DEFT, Wanda, RIA, SSU, "removal cost," "capacity") are named without explanation, so the reader has no context of what the methods are, or their purpose. The paper is not self contained.
    • Basic terms (unit, entry, frozen core) are defined only after they are used.
    • The text is dense with numbers and self-coined jargon (removal-value, drift-value, headroom rule, screening rule, annex, isolated/exposed core, acquisition, retention), which makes the main argument hard to extract. These terms are different from the one's used by the ML community like finetuning and catastrophic forgetting which makes it difficult to place the paper in current context.
    • The HOPE method is mentioned in the abstract, without any context or explanation of what it is.
  2. The central claim is not directly tested. The paper prescribes removal-value for isolated cores and drift-value for exposed ones. However, drift-value is never evaluated on an isolated core, and no isolated transformer setting exists. The experiments show that removal-value needs isolation, not that each score must be paired with its regime.

  3. The practical benefit over existing scores is unclear. Drift-value is statistically indistinguishable from empirical Fisher on all six language models at 40 epochs, and from SSU in the main setting. The 160-epoch advantage over Fisher holds only on one model and disappears when Fisher is periodically recomputed. In the vision experiment, the authors expansion baseline retains far better performance (0.725 vs 0.440).

  4. The gains are hard to attribute to the score. Freezing all attention accounts for a large share of the retention improvement. The selection-specific gain is the difference over a random mask, which shrinks considerably under less aggressive learning rates.

  5. Baselines are incomplete.

    • The CNN experiment compares selection only against DEFT.
    • Published methods (Wanda, RIA, SSU, Super-Tuning) are only used as scoring rules rather than run as proposed.
    • Related approaches such as MAS, Model-Dowser, S2FT and EWC-style regularization in transformers are not compared.
Questions for authors
  1. What is the exact research question that the authors are trying to answer and what is the proposed method for it. Is it the drift value scoring or the retention value scoring and how must one use it? And what is the expected practical benefit of the method over empirical Fisher?
  2. How does drift-value perform on the isolated VGG-8 core, and how does removal-value compare with Fisher there?
  3. It seems that acquisition always saturates to a relatively high score, what about evaluating the method on harder finetuning tasks which might distinguish the methods better on the finetuning performance and the retention?
  4. In Figure 4, what is the Frozen backbone + annex method, it is not discussed in detailed as a baseline and why the drift value score is relevant if this baseline is already performing so well?
7 comments
AuthorsOpenReview ↗

Part 1: question, method and connectivity

Thank you for your time and effort, and the constructive feedback showing us where the presentation needs a clearer entry point. You point out that frozen weights with drifting inputs are not really protected. That observation is the core of our paper, and the experiments you requested confirm it. Where the frozen neurons can be isolated, our removal-value sets a new state of the art it keeps 5.22 ± 0.51 points more than the best method at equal accuracy (Fig. 2, L193-197). In the comparison you asked for, it also leads seven further scores, Fisher and drift-value among them, by 6 to 20 points. Where the neurons cannot be isolated, as in transformers, drift-value matches Fisher, the strongest published score, at half the GPU memory, with one forward pass and no gradients.

W1 and Q1: What is the exact research question [...] what is the proposed method [...] how must one use it? Our research question is in the title and the introduction (L035): which parameters should we freeze, when updates elsewhere can still change their computation? Our proposed method is Guarded Freezing (L170-174). If the paths into the frozen neurons can be cut, select neurons by removal-value. If they cannot, as in transformers, freeze attention and select feed-forward entries by drift-value. The revision will state this in the abstract and will place the work more clearly in the parameter-isolation family of continual learning (De Lange et al., 2021).

Strength 1: [...] why the authors chose this specific (relatively outdated) CNN model. We chose VGG-8 as it is the benchmark of the state-of-the-art method we compare with (Mobahi and Bartlett, 2026, reproduced in Appendix Table 7), this network allows for incoming paths to be cut cleanly. DINOv3 and six language models cover modern transformers (Sections 5 and 6).

W1: [...] self-coined jargon [...] methods [...] named without explanation Retention, acquisition, frozen core and parameter isolation are established terms used in (Yang et al., 2024, Yamaguchi et al., 2026, Mobahi and Bartlett, 2026, De Lange et al., 2021). You are right with our own jargon, Removal-value, drift-value and the headroom rule as they name our contributions. The revision will define unit, entry and frozen core before first use and introduce HOPE, DEFT, Wanda, RIA and SSU in one sentence each.

Q2/W2 (new evidence): The central claim is not directly tested. [...] How does drift-value perform on the isolated VGG-8 core, and how does removal-value compare with Fisher there? Figure 2 tests the claim directly: the same cores with paths cut and with paths kept (L193-202). (new evidence): However, we ran your requested experiment in the setting of Figure 2, 40% Removal-value retains 0.252, drift-value 0.183 and Fisher 0.129, at equal SVHN accuracy (0.81 to 0.82). Removal-value keeps almost twice Fisher's retention. With paths kept, all three fall to 0.06 to 0.07. The reverse holds in language models, where paths stay open: removal-value cores do not beat random (Appendix Table 9) and drift-value does (Table 1). Residual streams make isolation impossible in transformers (L106-107), which is why the second score exists.

W5 (new evidence): The CNN experiment compares selection only against DEFT. The paper already provides more comparisons (Appendix Table 7). However, we added seven scores on the isolated core with the same VGG-8 setting: drift-value, source Fisher, target Fisher (FISH), RIA, SSU, Wanda and magnitude. Removal-value leads all seven, by 6 to 20 points of CIFAR-100 accuracy, at similar SVHN accuracy (0.80 to 0.82).

Part 2 covers practical benefit, baselines and harder tasks.

AuthorsOpenReview ↗

Part 2: practical benefit, baselines and tasks

W3/Q1 (new evidence): The practical benefit over existing scores is unclear. [...] over empirical Fisher? Our removal-value sets a new state of the art where isolation is possible. And, where it is not, drift-value achieves competitive retention at half the memory footprint. Drift-value needs one forward pass and no gradients. (new evidence): calibration peaks at 7.12 GiB against Fisher's 14.37 GiB and runs 2.6 times faster (0.371 s vs 0.970 s) on Qwen3-1.7B with the same 12 prompts and GPU. Its masks are more stable across calibration draws (Jaccard 0.89 vs 0.69, L292). Over long training, Fisher must be recomputed every 20 epochs, with backward passes each time, to reach what one drift-value mask computed once already holds (0.441 vs 0.443 on Qwen2.5-1.5B after 160 epochs, L1329-1331).

W4: The gains are hard to attribute to the score. Table 1 separates the two effects on Qwen3-1.7B, where the pretrained model scores 0.577. Full tuning on the new facts drops retention to 0.263, and attention freezing lifts it to 0.356. A random mask on top reaches 0.426 and drift-value reaches 0.502. This drift-value selection alone adds 0.076 ± 0.006, positive in all nine seeds (L251-252), and 0.253 ± 0.008 image accuracy when DINOv3 is adapted to point clouds (L399). At lower rates there is less forgetting to prevent, and Section 7 predicts the size of the gain from it.

W5: Baselines are incomplete. Part 1 adds seven scores to the CNN comparison. The paper already includes LoRA, HFT, adapters, expansion, EWC, head-only and BN-only tuning (Appendix Tables 6, 7, 13 and 14). Published scores run inside one protocol so that only the selection differs (L255-263). New evidence: EWC on Qwen3-1.7B in the protocol of Table 1, nine seeds, is on par with drift-value (0.012 ± 0.013 in favor of drift-value) and EWC needs gradients.

Q3 (new evidence): [...] acquisition always saturates [...] harder finetuning tasks The invented facts guarantee a task that no model has seen in pretraining. For a harder task we ran five language models tuned on Super-NaturalInstructions and scored by ROUGE-L on 119 unseen tasks, with retention measured as in the paper (worst accuracy over HellaSwag, ARC-Easy and SciQ). Freezing attention and the 35.2% of feed-forward entries with the highest drift-value beats full tuning on both axes. On Qwen3-1.7B, Qwen2.5-1.5B and Qwen3-0.6B at learning rate 1e-4, it keeps 0.04 to 0.10 more retention and 0.08 more ROUGE-L after one epoch, and 0.10 to 0.14 more retention and 0.10 to 0.12 more ROUGE-L after three epochs. In every model and rate, at both durations, it stays within 0.02 retention and 0.01 ROUGE-L of Fisher, without gradients. Acquisition no longer saturates: on Qwen3-1.7B, ROUGE-L spans 0.39 to 0.48 across policies, from 0.05 pretrained. Full tables follow in a separate comment.

Q4: [...] what is the Frozen backbone + annex method [...] why the drift value score is relevant? The annex (L373, Appendix C) widens (adds neurons) every feed-forward layer by 10% and freezes the whole backbone. It buys retention with extra parameters and memory, so it is a reference for practitioners who can grow the model, not a same-size competitor. Drift-value works inside the existing parameter budget.

We hope this helps you revisit your assessment, and we welcome further questions.

Additional results: instruction tuning, one epoch

Retention (minimum accuracy over commonsense, reasoning and knowledge)

Model, ratePretrainedFullAttention frozenRandomDrift-valueFisherSSU
Qwen3-1.7B, 1e-40.5400.498 ± 0.0070.521 ± 0.0070.560 ± 0.0070.539 ± 0.0070.533 ± 0.0050.556 ± 0.005
Qwen2.5-1.5B, 1e-40.6400.510 ± 0.0140.550 ± 0.0140.580 ± 0.0100.612 ± 0.0160.607 ± 0.0060.603 ± 0.006
Qwen3-0.6B, 1e-40.4500.377 ± 0.0120.397 ± 0.0080.430 ± 0.0080.438 ± 0.0090.443 ± 0.0150.443 ± 0.010
SmolLM2-1.7B, 1e-40.6500.658 ± 0.0040.673 ± 0.0140.678 ± 0.0030.663 ± 0.0030.665 ± 0.0000.673 ± 0.003
SmolLM2-360M, 1e-40.5350.510 ± 0.0050.515 ± 0.0080.513 ± 0.0070.517 ± 0.0060.513 ± 0.0070.513 ± 0.006
Qwen3-1.7B, 2e-50.5400.596 ± 0.0040.593 ± 0.0030.589 ± 0.0040.562 ± 0.0040.553 ± 0.0030.569 ± 0.005
Qwen2.5-1.5B, 2e-50.6400.632 ± 0.0030.642 ± 0.0070.630 ± 0.0030.635 ± 0.0030.630 ± 0.0030.632 ± 0.002
Qwen3-0.6B, 2e-50.4500.453 ± 0.0020.470 ± 0.0030.467 ± 0.0090.467 ± 0.0070.475 ± 0.0050.478 ± 0.009
SmolLM2-1.7B, 2e-50.6500.668 ± 0.0020.678 ± 0.0030.677 ± 0.0030.667 ± 0.0030.665 ± 0.0000.667 ± 0.003
SmolLM2-360M, 2e-50.5350.528 ± 0.0030.535 ± 0.0050.538 ± 0.0030.535 ± 0.0030.538 ± 0.0020.537 ± 0.002

Instruction quality (macro ROUGE-L on 119 unseen tasks)

Model, ratePretrainedFullAttention frozenRandomDrift-valueFisherSSU
Qwen3-1.7B, 1e-40.0520.394 ± 0.0040.440 ± 0.0040.467 ± 0.0040.475 ± 0.0040.481 ± 0.0040.480 ± 0.006
Qwen2.5-1.5B, 1e-40.0880.365 ± 0.0050.381 ± 0.0070.425 ± 0.0160.442 ± 0.0080.446 ± 0.0010.444 ± 0.000
Qwen3-0.6B, 1e-40.0520.351 ± 0.0040.394 ± 0.0050.425 ± 0.0020.434 ± 0.0030.438 ± 0.0040.435 ± 0.007
SmolLM2-1.7B, 1e-40.0630.429 ± 0.0040.440 ± 0.0070.443 ± 0.0090.445 ± 0.0090.449 ± 0.0100.450 ± 0.002
SmolLM2-360M, 1e-40.0560.376 ± 0.0050.386 ± 0.0080.396 ± 0.0050.397 ± 0.0020.393 ± 0.0040.396 ± 0.004
Qwen3-1.7B, 2e-50.0520.507 ± 0.0030.519 ± 0.0030.523 ± 0.0020.524 ± 0.0020.523 ± 0.0020.522 ± 0.001
Qwen2.5-1.5B, 2e-50.0880.494 ± 0.0100.509 ± 0.0060.519 ± 0.0070.523 ± 0.0100.523 ± 0.0090.521 ± 0.012
Qwen3-0.6B, 2e-50.0520.450 ± 0.0020.472 ± 0.0080.477 ± 0.0040.478 ± 0.0020.479 ± 0.0010.478 ± 0.007
SmolLM2-1.7B, 2e-50.0630.479 ± 0.0060.472 ± 0.0050.465 ± 0.0020.465 ± 0.0020.464 ± 0.0010.465 ± 0.001
SmolLM2-360M, 2e-50.0560.402 ± 0.0020.401 ± 0.0050.398 ± 0.0040.392 ± 0.0040.393 ± 0.0050.396 ± 0.005

Mean ± SE over three seeds. Pretrained is one evaluation. Every policy except full tuning freezes attention. Random, drift-value, Fisher and SSU also freeze 35.2% of feed-forward entries.

Reviewer_YCi2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Guarded Freezing, a connectivity-dependent selective-freezing framework. It uses removal-value to select coherent units when plastic-to-frozen paths are cut, and a forward-only drift-value score to select scalar feed-forward weight entries when residual-stream paths remain open. It evaluates VGG-8 transfer, invented-fact adaptation in language models, and DINOv3 point-cloud adaptation while measuring old-task retention and new-task acquisition.

Critical strengths

The paper compares drift-value with the relevant published scoring functions and openly reports the performance relative to Fisher and SSU. The drift-value approach is cheap, label-free, and forward-only.

Critical weaknesses

The paper is framed partly as an empirical characterization of what to freeze. In substance, however, it is a method for fine-tuning without losing performance on already-trained tasks, arguably a proper subset of continual learning, the literature of which is copious. The evaluation pits the authors' method against existing approaches. See how in Table 1, the best performing approaches in terms of retention are Fisher, drift-value, and SSU, respectively. (The acquisition scores in Table 1 have little variance across methods.)

A key problem with this presentation is that the amount of the feed-forward matrices that is frozen in each layer is fixed at 35.2%. In the continual learning literature, evaluations should be exacting, and in this case, a complete study of the efficacy of the authors' method would evaluate the approach across a suitable range of percentages and allow the percentage to vary across layers.

Questions for authors

Please help the reader understand the relative performance of your and other methods:

  • Run and report retention and acquisition versus freezing percentage for drift-value, random, Fisher, and SSU on Qwen3-1.7B with freezing percentages [10%, 20%, 30%, 40%, 50%].
  • Do the same as requested in the previous bullet item, but on the 1st ... Nth layer of the network, separately. Please report this in the same format as Table 1, including seeds and confidence.
4 comments

Thank you for your time and effort, and for asking for an exacting evaluation. We ran every experiment you requested on our worst-performing model, and the results of the paper hold. Freezing a weight does not protect it, because updates elsewhere still change what it computes. Our paper shows that this connectivity decides which freezing rule works. Where the frozen neurons can be isolated, our removal-value sets a new state of the art it keeps 5.22 ± 0.51 points more than the best method at equal accuracy (Fig. 2, L193-197). Where they cannot be isolated, as in transformers, drift-value is, as you note, cheap, label-free and forward-only. At all five budgets you requested it stays within 0.02 retention of Fisher, the strongest published score, at half the GPU memory.

W1: [...] framed partly as an empirical characterization [...] however, it is a method for fine-tuning [...] the best performing approaches in terms of retention are Fisher, drift-value, and SSU You are right, however we believe the empirical evaluation to be as valuable as the method, as this realization about the two regimes of isolation might help shape future methods. In our case we set a new state of the art with removal-value only where isolation is possible and are competitive with our drift-value where it isn't possible to isolate. Drift-value, unlike Fisher, needs no gradients (which translates to the calibration being faster and requiring less GPU memory). New evidence: drift-value calibration peaks at 7.12 GiB against Fisher's 14.37 GiB and runs 2.6 times faster (0.371 s vs 0.970 s) on Qwen3-1.7B with the same 12 prompts and GPU. Its masks are also more stable across calibration draws (Jaccard 0.89 vs 0.69, L292).

W2: [...] fixed at 35.2% [...] evaluate the approach across a suitable range of percentages and allow the percentage to vary across layers. We chose 35.2% for the headline tables as it is the share of updates that bf16 fine-tuning already discards (L249-250), which makes the comparison with full tuning fair. However, the paper already tests 20% and 50% on Qwen3-1.7B (L252-253), three budgets on the DINOv3 vision transformer (Appendix Table 6) and 40% and 70% on VGG-8 (Fig. 2). On Qwen3-1.7B, counts that vary per matrix, from 25.5% to 48.0%, give the same retention (Appendix Table 12).

Q1 (new evidence): [...] retention and acquisition versus freezing percentage [...] [10%, 20%, 30%, 40%, 50%]. We provide the evidence you requested: on Qwen3-1.7B (where drift-value performed worst, see Fig 3) in the protocol of Table 1: 40 epochs on the new facts at learning rate 1e-4, attention frozen, nine paired seeds per cell. The first column is the share of every feed-forward matrix that is frozen. Drift-value stays within 0.016 retention of Fisher at every budget, without gradients, and leads SSU at all budgets except 10%. Acquisition (recall of the new facts) is 0.985 to 0.994 for every method and budget.

Retention (worst-task accuracy)

FrozenRandomDrift-valueFisherSSU
10%0.384 ± 0.0120.436 ± 0.0070.452 ± 0.0110.444 ± 0.006
20%0.388 ± 0.0150.474 ± 0.0110.477 ± 0.0070.463 ± 0.009
30%0.408 ± 0.0090.497 ± 0.0080.502 ± 0.0060.477 ± 0.006
40%0.443 ± 0.0130.502 ± 0.0060.516 ± 0.0060.494 ± 0.007
50%0.471 ± 0.0120.518 ± 0.0060.526 ± 0.0080.504 ± 0.008

Mean ± SE, attention frozen in every column. Pretrained 0.577, full tuning 0.263, attention frozen alone 0.356 (Table 1).

Q2 (new evidence): Do the same [...] on the 1st ... Nth layer of the network, separately. Done, on the same model and protocol: we freeze 50% of the feed-forward entries of one block at a time, for each of the 28 blocks, four scores and nine seeds (1,008 runs). No single block carries the effect for any score. With attention frozen and no entries frozen, retention is 0.356 ± 0.011. Freezing half of one block leaves it there, whichever block and whichever score. Freezing half of every block lifts it to about 0.5. Protection comes from freezing across depth, which supports the uniform budget.

Where 50% of the feed-forward entries are frozenRandomDrift-valueFisherSSU
In one block only (mean ± SD over the 28 blocks)0.357 ± 0.0090.362 ± 0.0100.359 ± 0.0110.357 ± 0.012
In all 28 blocks (mean ± SE over nine seeds)0.471 ± 0.0120.518 ± 0.0060.526 ± 0.0080.504 ± 0.008

The other budgets agree: at 10%, 20%, 30% and 40% on blocks 1, 8, 15, 22 and 28 (three seeds), single-block retention stays between 0.29 and 0.37 for every score. The revised appendix will have the full per-block tables in the format of Table 1.

We hope this helps you revisit your assessment, and we welcome further questions.

AuthorsOpenReview ↗

Revised Manuscript

Thank you for your time and your criticism, we have revised the manuscript based on your feedback.

Our contribution is a principle for choosing what to freeze and a method built on it, Guarded Freezing. Freezing alone cannot preserve a network's performance while trainable parameters can still influence the frozen core, so the choice depends on connectivity. With the paths into the frozen core kept, no tested rule on VGG-8 separates from a random core (Figure 2a). With the paths cut, removal-value, a neuron's share of its layer's capacity, places the core across layers and retains 6 to 12 points more than DEFT at 40 to 70% frozen, while still learning the new task (Figure 2a, Table 9). With the paths open, an update disturbs a layer's output, and drift-value prices that disturbance from one forward pass. In contrast, Empirical Fisher, the strongest competitor in this regime, needs gradients. After 40 epochs drift-value stays within 1.7 points of it on six language models, and on Qwen3-1.7B its calibration pass is 2.6 times faster at half the memory.

The revised manuscript marks every new or changed passage, table and figure in blue. Its main changes are:

  • Where the paths can be cut, complete grids compare the selection rules at every share from 40 to 90% of neurons frozen on three CNNs (previously one), and a second transfer pair is added (Section 4, Tables 9 to 12). On VGG-8 from 50 to 70%, removal-value keeps the most old-task accuracy and reaches the new-task target in all 20 transfers. At 70%, the rules within the decision threshold of it reach that target in 2 of 20. On VGG-11 and VanillaNet-6 it retains the most at every share. Section 3.1 states removal-value as the share form of HOPE's removal cost, and Appendix A gives its exact properties.
  • Where the paths stay open, complete grids add MAS, Model-Dowser, S2FT and EWC on Qwen3-1.7B, and source and target Fisher on DINOv3 (Sections 5 and 6, Tables 13 and 14). On Qwen3-1.7B no rule retains more than drift-value beyond the decision threshold at any share, and empirical Fisher, which stays within it, needs a backward pass and twice the memory (Table 17). New tables freeze one block at a time and replace the invented facts with instruction tuning on Super-NaturalInstructions (Tables 15, 16, 18 and 19).
  • Clarity. The abstract states the question and the method, the introduction ends with a list of contributions, every term is defined at first use, and the method is placed in the parameter-isolation family of continual learning (Sections 1 and 2).

We would be happy to answer any additional questions that would help you revisit your assessment.

We thank the authors for their responses to our questions. We remain confident in our original rating and will leave it unchanged.

Thank you for reading our responses and the revised manuscript.

We would want to highlight that the revision contains both experiments that your review requested, on Qwen3-1.7B and in the format of Table 1:

  • Retention and acquisition at 10%, 20%, 30%, 40% and 50% frozen for drift-value, random, Fisher and SSU, with seeds and confidence (Appendix Table 13).
  • The same comparison with one block frozen at a time, for each of the 28 blocks (Appendix Tables 15 and 16).

The revision also places the method in the parameter-isolation family of continual learning (Sections 1 and 2), which answers the framing point of your review.

If a specific concern remains after these additions, we would be glad to address it. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.