- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Influence Spectrum framework and its core measure, the Parameter-space Influence Function (PIF), to characterize task-specific layer importance in large language models. PIF combines first-order gradient information with second-order directional curvature computed via Hessian-vector products to measure local loss sensitivity to parameter perturbations. The framework is validated on several downstream tasks across multiple open-source LLMs and benchmarks.
Critical strengths
The problem of identifying task-relevant layers is practically important. The use of Hessian-vector products avoids explicitly constructing or inverting the Hessian, which is computationally sensible. The paper covers a reasonably broad set of downstream applications.
Critical weaknesses
-
Limited novelty. Layer-wise importance has already been widely used in fine-tuning and model pruning. The first-order and second-order information used by PIF is not new. The paper also lacks a related-work discussion that clearly positions its contribution for readers outside this specific area.
-
Insufficient and unconvincing comparisons. For layer-masking, the paper does not compare against layer-wise model pruning methods. For layer-wise fine-tuning, the paper does not compare against the line of layer-wise fine-tuning methods such as LISA and IST.
-
Fine-tuning a small fixed set of layers may not yield good results. Layer importance can change across training stages due to training dynamics. Selecting a fixed subset of layers once and then fine-tuning only those layers is therefore not reliably justified.
-
The paper lacks a core narrative. PIF is applied to multiple tasks, but the paper does not establish a clear central story. It also lacks comparison with recent state-of-the-art methods in the evaluated settings.
DLP: dynamic layerwise pruning in large language models, ICML
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models, EMNLP
Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, NeurIPS
Questions for authors
please see weakness.