acceptodds
Under review as a conference paper at ICLR 2027

Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuning

Abstract

Parameter-efficient fine-tuning often selects trainable parameters before adaptation using architectural heuristics, without accounting for their varying importance during training. We introduce FisherAdapTune, which progressively selects parameter groups based on temporal drift in their Fisher information. Under a local Gaussian approximation, we bound the divergence between the fine-tuned posterior and pretrained prior by accumulated Fisher-weighted update costs, motivating curvature-aware selection. FisherAdapTune measures Jensen-Shannon distance between successive Fisher-value distributions and uses an adaptive threshold to freeze stabilized groups. Across VTAB-1k classification tasks, it achieves the highest macro Top-1 accuracy among the compared methods with a smaller average trainable set than full fine-tuning. Across four segmentation backbones, it maintains competitive in-distribution performance and improves zero-shot transfer in several settings. Its selections reveal architecture-dependent patterns where input, output, and normalization parameters can remain trainable as attention and MLP groups freeze at different rates. The results support Fisher structural drift as a task-dependent signal for allocating updates during adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Adaptive Fine-Tuning, Transfer Learning, Fisher Information, Automated Parameter Selection, Progressive Freezing, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ok5wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores the use of fisher information to selectively finetuning selected groups of paragraphs based on importance for PEFT.

The evaluations are performed on image classification tasks, using vtab-1k dataset.

The evaluations and comparisons show marginal / limited improvements compared to existing methods.

Critical strengths
  • Overall the paper is well motivated.

  • The evaluationsshows marginal / limited improvements on vtab-1k tasks.

  • The ablation studies and comparisons, as well as the provided figures, show some insights.

Critical weaknesses

Major:

  • One of the major and big issues of this work, is its very limited evaluations. The evaluations are only conducted on one dataset (vtab-1k), and focus only on image classification. The generalisationability of this work is unclear. The authors should evaluate on at least 3 datasets, following standard PEFT evaluations, otherwise the evaluations would be biased and do not provide any meaningful insights, given vtab-1k is also a pretty old dataset. zero-/few-shot evaluations should also be performed. In addition, the evaluations are mainly focusing on image classification, it is unclear if the proposed method would be helpful for eg object detection, segmentation, action recognition tasks, etc.

  • The novelty of this work, to me, appears to be unclear. The method section is organised in a quite sparse way, it is unclear why the modules are designed and arranged in this way. The lack of proper justifications and design principles make the novelty also unclear to assess. In addition, it would be better to follow the conventional maths notations in the whole paper to make the maths symbols and operations clear to readers.

  • Fig 2 is unclear to reviewer what are the main purposes of this figure, and what are the insights of the FisherAdpTune, and how the proposed model differs significantly from existing works. This work sounds more like applying standard fisher information to PEFT tasks.

  • The related work section is quite poorly organised: (i) it contains only one single paragraph, with two bold phrases highlighted; (ii) the review of existing works appear to be very limited, expecially the PEFT literature, the use of fisher information, the finetuning via importance scoring etc, are currently lacking; (iii) the review of existing works does not clearly show why the proposed method could contribute to the current literature, and how the proposed method differs from existing PEFT methods.

Minor:

  • Figure font sizes appear to be too small to read.

  • Table font sizes should be consistent in the whole paper.

  • The organisation of the paper needs significant improvements.

Questions for authors

Please see the weaknesses.

7 comments
AuthorsOpenReview ↗

Response to W1: Evaluation scope across datasets, tasks, and zero-shot transfer

W1: One of the major and big issues of this work, is its very limited evaluations. The evaluations are only conducted on one dataset (vtab-1k), and focus only on image classification. The generalisationability of this work is unclear. The authors should evaluate on at least 3 datasets, following standard PEFT evaluations, otherwise the evaluations would be biased and do not provide any meaningful insights, given vtab-1k is also a pretty old dataset. zero-/few-shot evaluations should also be performed. In addition, the evaluations are mainly focusing on image classification, it is unclear if the proposed method would be helpful for eg object detection, segmentation, action recognition tasks, etc.

Response:

This assessment is factually incorrect and does not reflect the content of the paper. Every point raised is already addressed in the submitted manuscript.

  • "Evaluations are only conducted on one dataset (vtab-1k)." This is not the case. VTAB-1k is a collection of 19 datasets across natural, specialized, and structured domains; all 19 are reported in the main paper (lines 341-353, Table 2) and per-task in the appendix. Segmentation adds four more datasets (OmniCrack30k, Concrete3k, Facade390, Road420) and four backbones. The "at least 3 datasets" is exceeded several times over.
  • "Focus only on image classification; unclear if helpful for segmentation." This is not the case. Segmentation is one of our two evaluation tasks, stated in the abstract, the contributions, Section 4 (Figure 7 of the submission / Figure 8 of the revised manuscript), and Appendix Tables 4 and 5.
  • "Zero-/few-shot evaluations should be performed." This is not the case. Segmentation models are trained on OmniCrack30k and evaluated zero-shot on three unseen datasets (Figure 7 and Table 4 of the original submission / Figure 8 and Table 4 of the revised manuscript). VTAB-1k is itself a few-shot benchmark (1,000 training examples per task) and the standard PEFT protocol used by all compared baselines.
  • "Generalizability is unclear." The paper covers two tasks, three architecture families (DINOv2 ViT, SAM2, SegFormer), 23 datasets, and zero-shot distribution shift. Detection and action recognition are not part of standard PEFT evaluation and are outside scope.

We ask the reviewer to reassess the paper on the basis of what it actually reports.

AuthorsOpenReview ↗

Response to W2: Method novelty, design rationale, and mathematical notation

W2: The novelty of this work, to me, appears to be unclear. The method section is organised in a quite sparse way, it is unclear why the modules are designed and arranged in this way. The lack of proper justifications and design principles make the novelty also unclear to assess. In addition, it would be better to follow the conventional maths notations in the whole paper to make the maths symbols and operations clear to readers.

Response:

This description of the method section does not accurately reflect the paper.

  • "Unclear why the modules are designed and arranged in this way." This is not the case. Section 3 is a single derivation chain. Section 3.1 demonstrates how Fisher information is related to the generalization error by deriving a PAC-Bayes bound where the KL term is bounded by accumulated Fisher-weighted update costs (Eq. 3 and Appendix A.1). Section 3.2 decomposes this cost across layers (Eq. 4) and shows that parameter groups that their Fisher geometry has stopped changing add cost without benefit (Eq. 6), motivating Fisher drift as an informative selection signal. Section 3.3 defines the scale-robust Fisher JS drift measure, verifies its heterogeneity across groups (Figures 4 and 5), and states Algorithm 1. Each step follows from the previous one.
  • "Lack of justification makes the novelty unclear." This is not the case. Table 1 and the contributions at lines 107-119 state it explicitly: most prior Fisher-based PEFT works (e.g., FISH-Mask) select parameters once by Fisher magnitude and they do not capture how a parameter's importance changes as adaptation proceeds (lines 73-74 of the manuscript). FisherAdapTune instead selects progressively by Fisher structural drift during training, motivated by a theoretical analysis. Tables 2 and 3 confirm this distinction, with Fisher drift outperforming Fisher magnitude and gradient-based signals.
  • "Follow conventional maths notation." The manuscript uses the standard Goodfellow et al. notation used by the default ICLR template, and every symbol is defined at first use. This assessment is factually incorrect.
AuthorsOpenReview ↗

Response to W3: Purpose of Figure 2 and distinction from Fisher-magnitude selection

W3: Fig 2 is unclear to reviewer what are the main purposes of this figure, and what are the insights of the FisherAdpTune, and how the proposed model differs significantly from existing works. This work sounds more like applying standard fisher information to PEFT tasks.

Response:

Please note this assessment is incorrect on both points.

  • "Unclear what the main purpose of Fig. 2 is." As stated in its caption and introductory text, Figure 2 conceptually illustrates FisherAdapTune’s core mechanism for a clearer understanding: periodically measuring the reorientation of each group's Fisher information via the Jensen-Shannon distance between consecutive Fisher distributions, keeping groups with high drift, and freezing stabilized groups. This is exactly the mechanism listed in the contributions and detailed in Section 3.3 and Algorithm 1. Its companion, Figure 3, shows the empirical observation that motivates it: Fisher drift is heterogeneous across layers.
  • "Sounds like applying standard Fisher information to PEFT." This is not the case. While existing Fisher-based PEFT methods (e.g., FISH-Mask) rank parameters by Fisher magnitude once, FisherAdapTune uses temporal change in the structure of the Fisher distribution to progressively determine which groups remain trainable. This criterion is motivated by our PAC-Bayes analysis (Eq. 6), distinguished from prior approaches in Table 1, and supported by comparisons against magnitude- and gradient-based selection in Table 3.

The insights the reviewer asks for are therefore already in the paper: adaptation importance is temporal and heterogeneous (Figures 3 to 5), Fisher structural drift, rather than Fisher magnitude, provides a signal for identifying them, and the trainable set is discovered by the method itself, emerging from the data and the architecture, rather than being fixed in advance by a prescribed budget or layer-freezing order.

Reviewer_epyJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FisherAdapTune. It progressively selects parameter groups based on temporal drift in their Fisher information. Under a local Gaussian approximation, the analysis links divergence from the pretrained prior to accumulated Fisher-weighted update costs. The method measures Jensen-Shannon distance between successive Fisher-value distributions and uses an adaptive threshold to freeze stabilized groups. Classification and segmentation experiments show competitive performance with fewer trainable parameters on average. The segmentation results also show improved zero-shot transfer in several settings.

Critical strengths

The paper offers a meaningful advance over fixed Fisher-based parameter selection. A parameter group can remain important after its adaptation has stabilized. Fisher magnitude alone does not capture this distinction. Temporal Fisher drift provides a way to assess the continued need for adaptation, which gives the selection criterion a clear methodological rationale. The signal ablations support its practical value. The paper presents the method clearly with experiments and well-designed figures and tables.

Critical weaknesses

The added value of progressive selection remains unclear. Drift differences emerge early and persist, but the experiments do not directly compare continued selection with one-time selection using the same drift signal.

Questions for authors
  1. Since drift differences emerge early and persist, what does continued selection add over selecting groups once after warm-up?
  2. Do any groups show low drift early but high drift later in full fine-tuning? What supports freezing these groups permanently?
  3. How did you choose the freezing thresholds? Did you use only source-domain validation data?
4 comments
AuthorsOpenReview ↗

Response to W1 & Q1: The role of continued selection beyond warm-up

W1: The added value of progressive selection remains unclear. Drift differences emerge early and persist, but the experiments do not directly compare continued selection with one-time selection using the same drift signal.

Q1: Since drift differences emerge early and persist, what does continued selection add over selecting groups once after warm-up?

Response:

Differences in Fisher drift do emerge early, however, Fisher drifts do not get stabilized at the same time across parameter groups. We show this on the SAM2-Tiny crack-segmentation experiment from the paper (Figure 6). The new figure, Figure 7 in the revised manuscript, plots the Fisher JS drift of every parameter group during FisherAdapTune fine-tuning. The figure shows a group frozen at 3k steps, while some decoder groups reach low drift only around 31–34k steps. The trainable parameter budget in Figure 6 shows how continued selection reduces the trainable set across 14 freezing decisions. Keeping the first selection fixed would leave 24.8M parameters trainable, compared with 0.27M at the end of continued selection (91$\times$ fewer).

Selecting once after warm-up therefore has to choose between two suboptimal options: (1) freeze only the groups that have stabilized by then, which leaves a large portion of parameters trainable (e.g., 24.8M parameters instead of 0.27M (91$\times$ more) in the discussed experiment), resulting in parameter inefficiency, or (2) freeze enough groups to match the final budget, which freezes the late-stabilizing groups ~25k steps before they finish adapting, resulting in potential performance degradation. Progressive selection avoids both by freezing each group when its own drift stops.

We have revised the manuscript accordingly, adding the new Figure 7 and a short discussion of why selection must continue beyond warm-up (lines 345–356 of the revised manuscript).

AuthorsOpenReview ↗

Response to Q2: Late-stage drift and the stability of freezing decisions

Q2: Do any groups show low drift early but high drift later in full fine-tuning? What supports freezing these groups permanently?

Response:

Thank you for asking this. Such groups are rare, and when they occur, their later drift generally remains below that of the active groups. We have added the supporting analysis below as a new section, Appendix F.2 (Pages 36–37), in the revised manuscript.

Direct test on full fine-tuning experiments. In our full fine-tuning runs nothing is frozen, so every group's drift can be followed to the end of training. On these experiments (DINOv2 on eight VTAB-1k tasks, and SAM2-Tiny on crack segmentation), we replayed the selection rule of Algorithm 1 with the paper's λ\lambda and freezing interval, recorded which groups it would have frozen and when, and kept following their actual drift afterwards. For each virtually frozen group we asked: (A) would its running-mean drift ever re-cross the pool threshold at a later decision, (B) is its drift over the last 20% of training above the final threshold, (C) is its late drift above the median of the groups that remained active.

Full fine-tuning traceGroups the rule would freeze(A) ever re-admitted(B) late drift >τfinal>\tau_{\text{final}}(C) late drift >> active-group median
DINOv2, 8 VTAB-1k tasks885 tensors (7.1–8.5M params/task)11 (1.2%)7 (0.8%)1 (0.1%)
SAM2-Tiny, segmentation task802 / 934 groups13 (1.6%)15 (1.9%)1 (0.1%)

The re-admitted cases are borderline groups with modest later drift (e.g., late JS 0.13–0.16 versus an active-group median of 0.18 on VTAB). Only one of 885 VTAB tensors and one of 802 SAM2 groups exceed the active-group median. Also, group rankings by mean drift over the first 10% and last 50% of training remain highly consistent (Spearman: 0.96–1.00 on VTAB; 0.995 on SAM2), indicating that groups with relatively low early drift generally retain low drift relative to their peers later.

What supports freezing permanently. Three complementary findings support this. (i) Sustained stabilization: the algorithm's selection rule (Algorithm 1) uses a running mean of EMA-smoothed JS, evaluated every nn steps against the active pool, reducing sensitivity to transient dips. The replay analysis shows that decisions rarely reverse. Also, the threshold parameter λ\lambda controls how conservatively groups are frozen (Figures 11 and 12 in the revised manuscript, and Tables 7 and 8). (ii) Theoretical motivation: under the stated assumptions, Eq. 6 and Appendix A motivate freezing stabilized groups to avoid further Fisher-weighted cost while preserving adaptation elsewhere. (iii) Empirical performance: FisherAdapTune achieves 61.62% versus 59.97% macro Top-1 across 19 VTAB-1k tasks with 7.11M versus 22.08M average trainable parameters (Table 2), lower validation loss than gradient-magnitude selection at matched checkpoints (Reviewer 7FmL, Q1), and competitive in-distribution segmentation with improved zero-shot transfer in several settings (Figure 8 of the revised manuscript, and Appendix Table 4).

AuthorsOpenReview ↗

Response to Q3: Threshold selection and source-domain validation

Q3: How did you choose the freezing thresholds? Did you use only source-domain validation data?

Response:

FisherAdapTune has a threshold hyperparameter, λ\lambda, which controls the adaptive cutoff τ=μ+λσ\tau=\mu+\lambda\sigma. The mean μ\mu and standard deviation σ\sigma are recomputed from the still-active groups’ Fisher JS drift at each freezing decision. Thus, λ\lambda specifies a relative criterion rather than a fixed JS threshold.

For VTAB-1k, we searched λ∈{−1.0,−0.8,−0.65,−0.5,−0.3}\lambda\in\{-1.0,-0.8,-0.65,-0.5,-0.3\} (Figure 11 in the revised manuscript, and Table 6). For segmentation, we tuned λ\lambda per backbone on OmniCrack30k using the source-domain validation F1–trainable-budget trade-off (Tables 7–8).

Thank you for addressing my comments. After reviewing your responses, I have decided to keep my original score unchanged.

Reviewer_7FmLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FisherAdapTune progressively freezes parameter groups using smoothed Jensen-Shannon distances between histograms of consecutive diagonal Kronecker Fisher estimates. A Gaussian posterior/prior analysis motivates curvature-aware selection. Experiments cover 19 VTAB-1k tasks with DINOv2 ViT-S/14 and crack segmentation with four backbones, including transfer to three datasets.

Critical strengths
  1. Flexible adaptation: The active set evolves without architectural changes or a prescribed layer-freezing order.

  2. Promising classification: Table 2 reports 61.62% macro accuracy versus 59.97% for full fine-tuning, using 7.11M versus 22.08M average trainable parameters.

  3. Informative diagnostics: This work documents computational trade-offs and architecture-dependent parameter lifetimes.

Critical weaknesses
  1. The freezing premise remains unvalidated. Eq. 3 bounds divergence, and Eq. 6 approximates update-cost drift; neither establishes that stable curvature implies negligible remaining adaptation benefit.

  2. Ablations confound signal and training effects. Table 3 uses 6.98M average trainable parameters for JS drift, versus 3.95M for random selection and 1.04M for Fisher-magnitude drift. Table 6 additionally changes weight decay (0.01 versus 0) and early-stopping patience (10 versus 15) between the reference and other configurations, weakening its sensitivity conclusions.

  3. Adaptive Fisher comparisons are incomplete. FISH-DIP [1] periodically refreshes Fisher masks during fine-tuning; FisherTune [2] uses domain-related Fisher selection with an evolving budget for segmentation. Temporal histogram drift is a distinct design, but its added value over existing adaptive Fisher selection is unresolved.

References:

[1] Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning. In EMNLP 2023.

[2] FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation. In CVPR 2025.

Questions for authors
  1. At matched checkpoints, does low JS drift predict smaller validation-loss benefits from continuing a group's updates, beyond what gradient magnitude predicts?
  2. Does the Table 3 advantage persist when alternative rankings follow the same active-parameter budget trajectory, including the initial fully trainable phase?
  3. How does drift-based selection compare with refreshed Fisher masks at matched budgets, and with FisherTune on a shared segmentation backbone?
4 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Freezing criterion and validation-loss comparison

W1: The freezing premise remains unvalidated. Eq. 3 bounds divergence, and Eq. 6 approximates update-cost drift; neither establishes that stable curvature implies negligible remaining adaptation benefit.

Q1: At matched checkpoints, does low JS drift predict smaller validation-loss benefits from continuing a group's updates, beyond what gradient magnitude predicts?

Response:

We agree that Eqs. 3 and 6 motivate the freezing criterion rather than prove that stable curvature implies negligible remaining adaptation benefit, as acknowledged in the abstract (lines 16–17) and Section 3 (lines 239–240 of the original submission / lines 243–244 of the revised manuscript). We therefore evaluate the criterion empirically by examining whether freezing groups with low Fisher JS drift preserves adaptation more effectively than freezing by gradient magnitude.

Specifically, we compared the validation losses of the Fisher JS drift and gradient RMS experiments from Table 3 at matched training checkpoints: steps 132, 144, 156, and 240 (epochs 11, 12, 13, and 20). These experiments use comparable trainable-parameter budgets, allowing us to assess how the two selection rules affect validation loss at the same training step.

TaskSignalStep 132Step 144Step 156Step 240
CIFAR-100Gradient RMS2.852.902.782.47
Fisher JS drift (ours)2.802.752.702.38
DTDGradient RMS1.831.821.761.91
Fisher JS drift (ours)1.751.711.741.66
Flowers-102Gradient RMS2.312.282.282.20
Fisher JS drift (ours)1.301.251.201.09
EuroSATGradient RMS0.660.660.710.63
Fisher JS drift (ours)0.630.650.610.61

Fisher JS drift yields lower validation loss on all four tasks at every reported checkpoint, consistent with its higher final accuracy in Table 3. These results support the practical value of curvature drift for deciding which groups to freeze relative to gradient magnitude.

AuthorsOpenReview ↗

Response to W2 & Q2: Adaptive budgets and ablation design

W2: Ablations confound signal and training effects. Table 3 uses 6.98M average trainable parameters for JS drift, versus 3.95M for random selection and 1.04M for Fisher-magnitude drift. Table 6 additionally changes weight decay (0.01 versus 0) and early-stopping patience (10 versus 15) between the reference and other configurations, weakening its sensitivity conclusions.

Response:

On the differing budgets in Table 3: The trainable budgets are outputs of the adaptive selection process, rather than being imposed as predefined inputs. All runs begin with the full model trainable. λ\lambda is tuned independently for each signal, and the budget each signal ends up with is its own output.

On the weight-decay and early-stopping patience in Table 6: We verified that these differences have only a marginal effect on the reference results and do not alter the sensitivity conclusions. In the original submission, the reference configuration used weight decay 10−210^{-2} and patience 10, whereas the alternative and sweep configurations used weight decay 0 and patience 15. Rerunning the reference configuration with the latter settings changes the four-task average accuracy from 78.05% to 78.20%, with a maximum absolute change of 1.06 percentage points on any task (CIFAR-100: 57.11 → 57.75, DTD: 71.65 → 70.59, SVHN: 87.67 → 88.48, EuroSAT: 95.76 → 96.00).

We have updated Table 6 and Figure 10 of the original submission (Figure 11 in the revised manuscript), so all configurations now use weight decay 0 and patience 15. The matched-setting results confirm that the reported sensitivity trends and conclusions are robust to these differences in training settings.

Q2: Does the Table 3 advantage persist when alternative rankings follow the same active-parameter budget trajectory, including the initial fully trainable phase?

Response:

Thank you for the question, however, we believe imposing our budget trajectory on the alternative signals would not be a valid control. The active-parameter budget trajectory is not an input to FisherAdapTune but one of its outputs: Fisher JS structural drift jointly discovers which groups to freeze and, consequently, when and how much budget to release. Transplanting this discovered trajectory onto another ranking would leak information produced by our signal into the baseline, so the comparison would no longer measure what the alternative signal can find on its own.

Instead, Table 3 gives every signal its own budget discovery under identical conditions: all runs (ours included) start fully trainable, and each alternative signal's λ (freezing threshold, which indirectly controls the parameter budget) was tuned. Most alternatives are unstable under this λ tuning: as the freezing threshold is relaxed they collapse to keeping (nearly) all parameters trainable, i.e., they degenerate toward full fine-tuning rather than discovering a selective schedule.

This behavior is verified in Figure 10 of the revised manuscript (Figure 9 of the original submission): Fisher structural drift is the only signal that shows a distinct separation between parameter groups, which is precisely what makes a stable, self-discovered budget trajectory possible.

AuthorsOpenReview ↗

Response to W3 & Q3: Adaptive Fisher comparisons and the added value of temporal drift

W3: Adaptive Fisher comparisons are incomplete. FISH-DIP [1] periodically refreshes Fisher masks during fine-tuning; FisherTune [2] uses domain-related Fisher selection with an evolving budget for segmentation. Temporal histogram drift is a distinct design, but its added value over existing adaptive Fisher selection is unresolved.

Response:

Please note the comparison with adaptive Fisher selection is not unresolved: our evaluation already addresses both adaptive freezing and Fisher-based parameter selection. We compare against a progressive/adaptive freezing method (AutoFreeze) and against Fisher-based selection (FISH-Mask, Child-Tuning$_D$) on VTAB-1k (Table 2) and on the shared segmentation backbones (Table 4), at comparable budgets.

In addition, Table 3 evaluates the added value of temporal Fisher drift within the same progressive-freezing framework. Its Fisher-magnitude variant recomputes Fisher importance at every freezing interval, providing a periodically refreshed Fisher criterion. Fisher structural drift achieves 82.65% versus 76.03% macro Top-1, a gain of 6.62 percentage points. This result supports the benefit of tracking changes in Fisher structure over periodically refreshing Fisher importance alone. We address the applicability of FISH-DIP and FisherTune separately in our response to the corresponding Question 3.

Q3: How does drift-based selection compare with refreshed Fisher masks at matched budgets, and with FisherTune on a shared segmentation backbone?

Response:

Refreshed Fisher masks at matched budgets. This comparison is addressed in the paper: the Fisher-magnitude signal in Table 3 recomputes Fisher importance at every freezing interval, providing a periodically refreshed Fisher mask within the same progressive-freezing framework. At comparable trainable budgets, Fisher structural drift outperforms the Fisher-magnitude, which supports the added value of tracking temporal changes in Fisher structure over repeatedly selecting by current Fisher importance. FISH-DIP's specific contribution beyond this criterion is a sample-feedback loop for low-resource NLP sequence labeling: it refreshes a fixed 1% mask every ~100 steps from the most-regressing training samples of a T5 model. Its selection signal (current Fisher magnitude) is the one our ablation already tests.

FisherTune on a shared segmentation backbone. We believe this comparison would conflate differences in task objectives, Fisher estimation, and budget schedules, rather than isolate the contribution of the selection signal. FisherTune is a domain-generalization framework for segmentation: its Domain-Related FIM requires simulated unseen-domain data (feature-statistic perturbations of source images) plus variational estimation, and its schedule expands the trainable set toward a preset maximum (δ_min → δ_max) to protect generalization under domain shift. FisherAdapTune addresses a different problem, general progressive parameter selection during fine-tuning, using only the task's own training data, no domain-shift simulation, and a shrinking, self-discovered budget. On a shared backbone the two methods would differ in objective (DGSS robustness vs. in-task adaptation), in the data their Fisher estimates consume, and in budget direction, so the outcome would not attribute differences to the selection signal.

We have highlighted this distinction in the Related Work section (lines 134–140 of the revised manuscript).

Thank you for the rebuttal and clarifications. I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the Responses

We thank all the reviewers for their time and feedback. The reviews raised four common concerns, which we summarize below with the changes made. The revisions are marked in blue in the revised manuscript.

1. Whether freezing low-drift groups is justified, and whether it should be permanent (Reviewers 7FmL and epyJ). We provided a comparison of validation loss at matched checkpoints in our response, where Fisher structural drift gives lower loss than gradient-magnitude selection on all four tasks. We also added a replay of our selection rule on full fine-tuning runs (new Appendix F.2), which shows that fewer than 2% of the groups it would freeze are ever re-admitted.

2. The added value of progressive selection over one-time selection (Reviewers epyJ and 7FmL). We added Figure 7 and a discussion showing that parameter groups stabilize at very different times, so a single selection after warm-up either leaves many parameters trainable or freezes groups that are still adapting.

3. Consistency of the ablation settings (Reviewer 7FmL). We reran the reference configuration with the same weight decay and early-stopping patience as the other configurations and updated Table 6 and Figure 11 accordingly. The marginal changes confirm that the reported sensitivity trends persist under identical training settings, leaving our conclusions unchanged.

4. Positioning relative to prior Fisher-based and PEFT methods, and the scope of the evaluation (Reviewers 7FmL and ok5w). We extended the Related Work section to include FISH-DIP and a clearer distinction from FisherTune. We also refer to the sections, tables, and figures in the original submission that already provide the requested evaluations: all 19 VTAB-1k tasks, semantic segmentation on four backbones, and zero-shot transfer to three unseen datasets.

Detailed responses to each reviewer follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.