acceptodds
Withdrawn from ICLR 2027

The Influence Spectrum: The Power of Layer-Wise Importance in Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The black-box nature of Large Language Models (LLMs) poses significant challenges to their transparency and controllability. Understanding which layers are most relevant to a given task provides a practical basis for interpreting LLM behavior and enabling effective task adaptation. We propose the influence spectrum framework to characterize task-specific layer importance in LLMs. At its core, the Parameter-space Influence Function (PIF) measures local loss sensitivity to parameter perturbations through first-order gradient information and second-order directional curvature. Using Hessian-vector products (HVPs), PIF condenses massive second-order operations into just two sequential gradient evaluations. Layer-masking experiments validate the diagnostic value of the resulting influence spectra, while extensive evaluations across diverse open-source LLMs and benchmarks demonstrate the utility of our framework in Parameter-Efficient Fine-Tuning (PEFT), continual learning, dynamic model defense, and knowledge editing.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Layer Importance, Influence Spectrum, Parameter-space Influence Function, Hessian-Vector Product, Model Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_w7smOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Influence Spectrum framework and its core measure, the Parameter-space Influence Function (PIF), to characterize task-specific layer importance in large language models. PIF combines first-order gradient information with second-order directional curvature computed via Hessian-vector products to measure local loss sensitivity to parameter perturbations. The framework is validated on several downstream tasks across multiple open-source LLMs and benchmarks.

Critical strengths

The problem of identifying task-relevant layers is practically important. The use of Hessian-vector products avoids explicitly constructing or inverting the Hessian, which is computationally sensible. The paper covers a reasonably broad set of downstream applications.

Critical weaknesses
  1. Limited novelty. Layer-wise importance has already been widely used in fine-tuning and model pruning. The first-order and second-order information used by PIF is not new. The paper also lacks a related-work discussion that clearly positions its contribution for readers outside this specific area.

  2. Insufficient and unconvincing comparisons. For layer-masking, the paper does not compare against layer-wise model pruning methods. For layer-wise fine-tuning, the paper does not compare against the line of layer-wise fine-tuning methods such as LISA and IST.

  3. Fine-tuning a small fixed set of layers may not yield good results. Layer importance can change across training stages due to training dynamics. Selecting a fixed subset of layers once and then fine-tuning only those layers is therefore not reliably justified.

  4. The paper lacks a core narrative. PIF is applied to multiple tasks, but the paper does not establish a clear central story. It also lacks comparison with recent state-of-the-art methods in the evaluated settings.

DLP: dynamic layerwise pruning in large language models, ICML

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models, EMNLP

Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, NeurIPS

Questions for authors

please see weakness.

Reviewer_KhdfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposed the influence spectrum framework to characterize task-specific layer importance in LLMs. Its core method uses a second-order Taylor expansion computed efficiently with a Hessian-vector product. The paper then demonstrates the effectiveness of its framework on various downstream tasks, which include selective finetuning, continual learning, knowledge editing, and jailbreak attacks.

Critical strengths
  1. I believe the paper is technically solid, and I do not see obvious technical flaws.
  2. The evaluation is fairly comprehensive, as it includes multiple downstream tasks.
Critical weaknesses
  1. The high-level motivation of this work needs further clarification. Traditional data influence is well motivated because each counterfactual evaluation would require leaving out one data point and retraining the model. With thousands or even millions of data points, these evaluations are simply infeasible. However, a typical LLM has much fewer layers. For example, in the fine-tuning experiments, one could conduct a short pilot run applying LoRA to each layer, then select the best-performing layers for a longer run. Such a procedure may not be prohibitively expensive. Without comparing against this direct selection baseline, the practical advantage of the proposed influence measure remains unclear, weakening the paper’s contribution.
  2. The overall technical contribution appears limited. The core method is basically a standard technique. Although I understand that applying this sensitivity analysis to the parameter space is a novel point, I feel like the authors need more justification and distinctions from traditional techniques.
Questions for authors

See weakness.

Reviewer_CxnMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work proposes a parameter influence method that is based on the estimation of the loss change under layer weight perturbation in a fixed direction, using a second-order Taylor expansion. Treating the perturbation direction as constant facilitates fast Hessian-vector product computation. The proposed influence estimators PIFl(1)PIF^{(1)}_l and PIFl(2)PIF^{(2)}_l are tested broadly on different applications: layer masking, fine-tuning, continual learning, model defense, knowledge editing.

Critical strengths
  1. The work considers many applications and how the method performs in terms of efficiency.
Critical weaknesses
  1. It seems that the influence function part of Section 2 is stated only for context and is never referenced further.

  2. The method itself is relatively simple, but Section 3.1 does not take into consideration the η\eta parameter. When the two PIF components are used together, η\eta becomes a hyperparameter which may significantly change the outcome of the layer influence estimation.

  3. Possible novelty issue. The authors position their work as a new influence method, but the classic approaches of Optimal Brain Damage (OBD) and, further, Optimal Brain Surgeon (OBS) already explored the idea of a second-order Taylor expansion of the loss change. At least a compare and contrast is expected.

  4. In Table 1, PIF does not dominate the other methods, the text and the table are formatted as though it does.

  5. In the application ablations, the authors do not consider the most recent layer influence estimation methods, gradient-based like LayerIF for instance (even though it is mentioned in the literature review), or activation-based like ShortGPT's Block Influence.

Questions for authors
  1. What η\eta values were used in the experiments? Since PIF(1)PIF^{(1)} and PIF(2)PIF^{(2)} scale as η\eta and η2\eta^2, the Combined ranking in Figure 2 depends on η\eta. An ablation over η\eta for the cases where both PIF terms are used for ranking would be helpful.

  2. How are the PIF scores aggregated across the task samples zz used for scoring? Do you average the per-sample PIF (with or without the absolute value), or compute PIF once from the averaged gradient and Hessian of the task loss?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.