- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper explores the use of fisher information to selectively finetuning selected groups of paragraphs based on importance for PEFT.
The evaluations are performed on image classification tasks, using vtab-1k dataset.
The evaluations and comparisons show marginal / limited improvements compared to existing methods.
Critical strengths
-
Overall the paper is well motivated.
-
The evaluationsshows marginal / limited improvements on vtab-1k tasks.
-
The ablation studies and comparisons, as well as the provided figures, show some insights.
Critical weaknesses
Major:
-
One of the major and big issues of this work, is its very limited evaluations. The evaluations are only conducted on one dataset (vtab-1k), and focus only on image classification. The generalisationability of this work is unclear. The authors should evaluate on at least 3 datasets, following standard PEFT evaluations, otherwise the evaluations would be biased and do not provide any meaningful insights, given vtab-1k is also a pretty old dataset. zero-/few-shot evaluations should also be performed. In addition, the evaluations are mainly focusing on image classification, it is unclear if the proposed method would be helpful for eg object detection, segmentation, action recognition tasks, etc.
-
The novelty of this work, to me, appears to be unclear. The method section is organised in a quite sparse way, it is unclear why the modules are designed and arranged in this way. The lack of proper justifications and design principles make the novelty also unclear to assess. In addition, it would be better to follow the conventional maths notations in the whole paper to make the maths symbols and operations clear to readers.
-
Fig 2 is unclear to reviewer what are the main purposes of this figure, and what are the insights of the FisherAdpTune, and how the proposed model differs significantly from existing works. This work sounds more like applying standard fisher information to PEFT tasks.
-
The related work section is quite poorly organised: (i) it contains only one single paragraph, with two bold phrases highlighted; (ii) the review of existing works appear to be very limited, expecially the PEFT literature, the use of fisher information, the finetuning via importance scoring etc, are currently lacking; (iii) the review of existing works does not clearly show why the proposed method could contribute to the current literature, and how the proposed method differs from existing PEFT methods.
Minor:
-
Figure font sizes appear to be too small to read.
-
Table font sizes should be consistent in the whole paper.
-
The organisation of the paper needs significant improvements.
Questions for authors
Please see the weaknesses.
7 comments
Response to W2: Method novelty, design rationale, and mathematical notation
W2: The novelty of this work, to me, appears to be unclear. The method section is organised in a quite sparse way, it is unclear why the modules are designed and arranged in this way. The lack of proper justifications and design principles make the novelty also unclear to assess. In addition, it would be better to follow the conventional maths notations in the whole paper to make the maths symbols and operations clear to readers.
Response:
This description of the method section does not accurately reflect the paper.
- "Unclear why the modules are designed and arranged in this way." This is not the case. Section 3 is a single derivation chain. Section 3.1 demonstrates how Fisher information is related to the generalization error by deriving a PAC-Bayes bound where the KL term is bounded by accumulated Fisher-weighted update costs (Eq. 3 and Appendix A.1). Section 3.2 decomposes this cost across layers (Eq. 4) and shows that parameter groups that their Fisher geometry has stopped changing add cost without benefit (Eq. 6), motivating Fisher drift as an informative selection signal. Section 3.3 defines the scale-robust Fisher JS drift measure, verifies its heterogeneity across groups (Figures 4 and 5), and states Algorithm 1. Each step follows from the previous one.
- "Lack of justification makes the novelty unclear." This is not the case. Table 1 and the contributions at lines 107-119 state it explicitly: most prior Fisher-based PEFT works (e.g., FISH-Mask) select parameters once by Fisher magnitude and they do not capture how a parameter's importance changes as adaptation proceeds (lines 73-74 of the manuscript). FisherAdapTune instead selects progressively by Fisher structural drift during training, motivated by a theoretical analysis. Tables 2 and 3 confirm this distinction, with Fisher drift outperforming Fisher magnitude and gradient-based signals.
- "Follow conventional maths notation." The manuscript uses the standard Goodfellow et al. notation used by the default ICLR template, and every symbol is defined at first use. This assessment is factually incorrect.
Response to W3: Purpose of Figure 2 and distinction from Fisher-magnitude selection
W3: Fig 2 is unclear to reviewer what are the main purposes of this figure, and what are the insights of the FisherAdpTune, and how the proposed model differs significantly from existing works. This work sounds more like applying standard fisher information to PEFT tasks.
Response:
Please note this assessment is incorrect on both points.
- "Unclear what the main purpose of Fig. 2 is." As stated in its caption and introductory text, Figure 2 conceptually illustrates FisherAdapTune’s core mechanism for a clearer understanding: periodically measuring the reorientation of each group's Fisher information via the Jensen-Shannon distance between consecutive Fisher distributions, keeping groups with high drift, and freezing stabilized groups. This is exactly the mechanism listed in the contributions and detailed in Section 3.3 and Algorithm 1. Its companion, Figure 3, shows the empirical observation that motivates it: Fisher drift is heterogeneous across layers.
- "Sounds like applying standard Fisher information to PEFT." This is not the case. While existing Fisher-based PEFT methods (e.g., FISH-Mask) rank parameters by Fisher magnitude once, FisherAdapTune uses temporal change in the structure of the Fisher distribution to progressively determine which groups remain trainable. This criterion is motivated by our PAC-Bayes analysis (Eq. 6), distinguished from prior approaches in Table 1, and supported by comparisons against magnitude- and gradient-based selection in Table 3.
The insights the reviewer asks for are therefore already in the paper: adaptation importance is temporal and heterogeneous (Figures 3 to 5), Fisher structural drift, rather than Fisher magnitude, provides a signal for identifying them, and the trainable set is discovered by the method itself, emerging from the data and the architecture, rather than being fixed in advance by a prescribed budget or layer-freezing order.
Response to W1: Evaluation scope across datasets, tasks, and zero-shot transfer
Response:
This assessment is factually incorrect and does not reflect the content of the paper. Every point raised is already addressed in the submitted manuscript.
We ask the reviewer to reassess the paper on the basis of what it actually reports.