acceptodds
Under review as a conference paper at ICLR 2027

Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

Abstract

Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability. We introduce Weight-Adjusted Gradients (WAG), a simple yet effective approach for estimating parameter importance that explicitly captures the interaction between model weights and first-order gradient information and identifies parameters that disproportionately influence model behavior, such as those responsible for collapse phenomena in LLMs. Across a range of models and settings, we show that WAG surfaces a tiny but critical subset of parameters (≤ 0.5 parts per million or 0.00005% of model size) whose modification leads to dramatic degradation in performance, indicating a novel failure mode. These findings also reveal a previously underexplored interplay between weights and gradients, suggesting that parameter importance cannot be fully understood through either signal alone. We demonstrate the practical utility of WAG across several diverse applications, such as expert allocation in Mixture-of-Experts (MoE) architectures, targeted unlearning, mixed-precision quantization, and layer selection for knowledge editing. In sum, WAG can serve as a unified approach for analyzing, debugging, and controlling LLMs, and opens new directions for principled parameter-level interpretation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter importance, model collapse, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eP7ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Weight-Adjusted Gradients method which utilizes zero-order and first-order information to identify the important weights in the LLM. The authors provide a series of downstream task to show the application and effectiveness of the Weight-Adjusted Gradients method.

Critical strengths
  • The paper is clear and easy to follow.
  • The proposed method is concise.
  • The experiment results are comprehensive and the findings about the impact of important weights in LLM is insightful.
Critical weaknesses
  • The Weight-Adjusted Gradients methodis mathematically similar to the existing Taylor-based saliency measure, which limits the novelty.

  • The improvement on MoE expert allocation is limited and not statistically significant, also for kjnowledge editing.

  • The targeted unlearning experiments do not compare WAG-based layer selection with random layer selection.

Questions for authors
  • Could the authors add some baselines such as random layer selection in targeted unlearning experiments?

  • Could the authors provide more statistically significant experiment results on MoE expert allocation and knowledge editing?

5 comments
AuthorsOpenReview ↗

Response to Reviewer eP7R [1/3]

Dear Reviewer eP7r,
Thank you for your feedback and for the effort spent on our work. We provide clarifications below. In tables, the best is bold and second best is highlighted.


W1. The Weight-Adjusted Gradients methodis mathematically similar to the existing Taylor-based saliency measure, which limits the novelty.

Response: Novelty over Taylor-based Saliency

Below, we discuss how the Weight-Adjusted Gradients (WAG) method differs from the existing Taylor-based saliency measures mathematically, none of which directly measures or observes the impact of WAG.

  • OBD [2], a second-order Taylor-based saliency measure, excludes the first-order relation between weight and gradient and only considers the second-order Taylor approximation.
  • Lee et al. [3], a first-order saliency measure, ignores the WAG relation and instead uses ∂L∂ci\frac{\partial L}{\partial c_i} as the saliency measure for pruning, where ci∈0,1c_i \in \\{0,1\\} indicates whether the network is connected to the weight or not.
  • Molchanov et al. [1] consider the loss LL to be a function of the activation hih_i, where the removal of a parameter causes the activation to be zero (hi=0h_i = 0). Hence, the first-order Taylor series expansion of their loss leads to the product of the output activation and the gradient of the loss with respect to the activation, i.e., hi∂L∂hih_i \frac{\partial L}{\partial h_i}.
  • WAG, −θi∂L∂θi-\theta_i \frac{\partial L}{\partial \theta_i}, as proved in Theorem 3.2 and Theorem 3.3, identifies the parameters to which the loss is most sensitive under multiplicative perturbations. However, existing Taylor-based approaches try to minimize the change in loss due to the removal of a parameter.

Therefore, none of the previous works directly explores the WAG term, and they do not study the direct impact of the most sensitive parameters identified by WAG.

AuthorsOpenReview ↗

Response to Reviewer eP7R [2/3]

W2. The improvement on MoE expert allocation is limited and not statistically significant, also for kjnowledge editing.

Response: Statistical Significance on MoE and Knowledge Editing

We thank the reviewer for raising this concern about the statistical significance of the improvements on MoE expert allocation and knowledge editing. We show below that the gains of WAG are statistically significant. On MoE, although the gains appear to be small, the gain of the best WAG variant over the best baseline is statistically significant for both models. On knowledge editing, |WAG| performs statistically better than CMA across all datasets for Gemma3, and is not statistically worse than CMA in 6 of the 7 tested settings:

  • MoE expert allocation: For simplicity, we consider the hypothesis to be the best-performing WAG variant vs. the best baseline. We conduct a one-sided paired t-test between the performance of the best baseline, MoLA Hourglass, and the best WAG variant, |WAG|, for the LLaMA3.2-3B model. For the Gemma3-12B model, the test is conducted between the performance of MoLA Rectangle and WAGLow\text{WAG}_\text{Low}. Each pair is one test sample scored by both methods, where the score is that sample's accuracy averaged over the 3 seeds. The samples are pooled across all datasets (MRPC, CoLA, and ScienceQA), giving 4,992 pairs per model. In both cases, the p-value is below 0.05, indicating that the performance gain of the best WAG variant over the best baseline is statistically significant:

One-sided paired t-test: best WAG variant vs. best baseline

ModelNull Hypothesisp-value
LLaMA3.2-3B|WAG| is not better than MoLA Hourglass3.5×10⁻⁷ < 0.05
Gemma3-12BWAGLow\text{WAG}_\text{Low} is not better than MoLA Rectangle2.6×10⁻³ < 0.05
  • Small gains are typical for MoE expert allocation: In prior work, LayerIF [4] reports that its best configuration improves over the best baseline by 0.34 points on Gemma-7B on average.

  • Knowledge editing: We conduct a similar one-sided sample-wise paired t-test based on the Overall metric. The paired t-test shows that across all datasets for Gemma3, |WAG|'s performance gain over CMA is statistically significant with p<0.05. On the other hand, CMA performs statistically better than |WAG| in only one of the 7 tested settings (WikiBio on LLaMA3.2), covering Gemma3 and LLaMA3.2 on ZSRE, WikiCounterFact, WikiBio, and WikiRecent. On WikiRecent with LLaMA3.2, both methods select the same layer, so no test applies.

One-sided paired t-test, null hypothesis: |WAG| is not better than CMA

DatasetModelp-value
ZSRELLaMA3.2-3B1.4×10⁻¹ ≥ 0.05
Gemma3-12B5.6×10⁻³⁵ < 0.05
WikiCounterFactLLaMA3.2-3B2.8×10⁻¹ ≥ 0.05
Gemma3-12B8.9×10⁻³⁷ < 0.05
WikiBioLLaMA3.2-3B9.7×10⁻¹ ≥ 0.05
Gemma3-12B2.0×10⁻³⁰ < 0.05
WikiRecentLLaMA3.2-3Bn/a (same layer selected)
Gemma3-12B3.8×10⁻² < 0.05

One-sided paired t-test, null hypothesis: CMA is not better than |WAG|

DatasetModelp-value
ZSRELLaMA3.2-3B8.6×10⁻¹ ≥ 0.05
Gemma3-12B1.0 ≥ 0.05
WikiCounterFactLLaMA3.2-3B7.2×10⁻¹ ≥ 0.05
Gemma3-12B1.0 ≥ 0.05
WikiBioLLaMA3.2-3B3.0×10⁻² < 0.05
Gemma3-12B1.0 ≥ 0.05
WikiRecentLLaMA3.2-3Bn/a (same layer selected)
Gemma3-12B9.6×10⁻¹ ≥ 0.05
AuthorsOpenReview ↗

Response to Reviewer eP7R [3/3]

W3. The targeted unlearning experiments do not compare WAG-based layer selection with random layer selection.

Response: Random Layer Selection Baseline

From the table below, we see that WAG criteria select better layers for targeted unlearning than random selection. The three WAG-based approaches are among the top 3 for targeted unlearning for both metrics.

### Targeted unlearning layer selection: WAG vs. random

MethodForget TRKS Forget
GradDiff (all layers)0.497±0.0560.588±0.014
GradDiff + Random0.475±0.0410.583±0.008
GradDiff + WAGLow\text{WAG}_\text{Low}0.546±0.0430.607±0.015
GradDiff + WAGHigh\text{WAG}_\text{High}0.541±0.0230.599±0.013
GradDiff + |WAG|0.589±0.0060.613±0.007

Q1. Could the authors add some baselines such as random layer selection in targeted unlearning experiments?

Response: Random Layer Selection Baseline

WAG-based layer selections outperform random layer selection for the task of targeted unlearning, as shown in our response to W3.



Q2. Could the authors provide more statistically significant experiment results on MoE expert allocation and knowledge editing?

Response: Statistical Significance Results

Statistical significance tests show that |WAG| performs statistically better than CMA across all datasets for Gemma3, and CMA performs statistically better than |WAG| only on the WikiBio dataset for the LLaMA3.2 model. We provide detailed statistical significance results in response to W2.


References:

[1] Pruning Convolutional Neural Networks for Resource Efficient Inference, ICLR 2017
[2] Optimal Brain Damage, NeurIPS 1989
[3] SNIP: Single-shot Network Pruning based on Connection Sensitivity, ICLR 2019
[4] LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions, NeurIPS 2025

Reviewer_C2qyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Weight-Adjusted Gradients (WAG) is a parameter importance estimation metric that combines zeroth-order parameter magnitude with first-order gradient information via direct multiplication.

Critical strengths

S1: This work investigates the relationship and impact between parameter magnitudes and gradient information during parameter importance estimation, which provide some inspiration for the design of parameter importance estimation metrics.

S2: Several empirical analyses across various application scenarios provide strong evidence for the effectiveness of the parameter importance estimation method WAG.

Critical weaknesses

W1: The motivation for combining parameter magnitude and gradient information during parameter importance estimation is not fully discussed. It seems that current logic is empirical evidence show that the two facts can give better estimation when they are used together. But why? The underlying reason for the helpfulness of WAG is stilling waiting exploration. Without such insights, only changing evaluation metrics make the work not informative enough.

W2: Only limited and classical metrics used to estimate parameter importance are compared in experiments. A lot of recent works are not listed, like integrated gradients methods[1], OBD [2], and so on.

[1] Axiomatic Attribution for Deep Networks, ICML 2017

[2] Optimal Brain Damage, NeurIPS 2026

Questions for authors

Q1: Why do you choose to using multiplication to combine the parameter magnitude and the first-order gradient. Have you tried other combination methods such as add or others?

Q2: What’s the difference between WAG and AdaLoRA? The calculation formulas for these two parameter importance estimation methods look the same.

Q3: Will integrating the second-order gradient information bring more benefits to WAG? There are already many good ways to approximate second-order gradient during parameter importance estimation for modern LLMs such as Model Shapley. I think high computational cost or approximation error should not be the obstacle.

[1] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR 2023

[2] MODEL SHAPLEY: Find Your Ideal Parameter Player via One Gradient Backpropagation, NeurIPS 2025

5 comments
AuthorsOpenReview ↗

Response to Reviewer C2qy [1/3]

Dear Reviewer C2qy,
Thank you for your feedback and for the effort spent on our work. We provide clarifications below. In tables, the best is bold and second best is highlighted.


W1. The motivation for combining parameter magnitude and gradient information during parameter importance estimation is not fully discussed. It seems that current logic is empirical evidence show that the two facts can give better estimation when they are used together. But why? The underlying reason for the helpfulness of WAG is stilling waiting exploration. Without such insights, only changing evaluation metrics make the work not informative enough.

Response: Motivation for Combining Weight and Gradient

The reviewer notes that the current logic appears to rely on empirical evidence that weight and gradient give a better estimate when used together. We would like to clarify that our motivation is not only empirical but also theoretical, established through Theorems 3.2 and 3.3: combining parameter magnitude and gradient allows WAG to measure parameter sensitivity to relative perturbation. In contrast, the gradient alone measures the loss change with respect to an additive perturbation, which is sensitive to parameter re-scaling [8], and the parameter weight alone does not directly relate to the change in loss [9]. We provide this theoretical motivation in detail below:

  • Theorem 3.2 (First-Order Multiplicative Sensitivity) establishes that, for multiplicative perturbations of the form θi′=θi(1+ϵi)\theta_i' = \theta_i(1+\epsilon_i), where ϵi\epsilon_i are sufficiently small, the corresponding first-order change in loss satisfies: ℓ(z;θ′)−ℓ(z;θ)=−∑i=1dϵi,WAGi+O(∣ϵ∣2).\ell(z;\theta') - \ell(z;\theta) = -\sum_{i=1}^{d} \epsilon_i \\, \mathrm{WAG}_i + O(\\|\epsilon\\|^2).

  • Theorem 3.3 (WAG Characterizes the Most Sensitive Log-Scale Direction) establishes that, letting w=(WAG1,…,WAGd)∈Rdw = (\mathrm{WAG}_1, \ldots, \mathrm{WAG}_d) \in \mathbb{R}^d denote the WAG vector, among all multiplicative perturbations satisfying ∣ϵ∣2=r\\|\epsilon\\|_2 = r, the maximal first-order increase in loss is r∣w∣2r\\|w\\|_2. Thus, WAG identifies the direction of multiplicative perturbation that produces the greatest first-order increase in loss under a fixed perturbation budget. Masking parameters corresponds directly to the multiplicative perturbation from 1⋅θi1 \cdot \theta_i to 0⋅θi0 \cdot \theta_i, where the perturbation budget is fixed for every parameter, i.e., 1→01 \rightarrow 0.

  • Why the gradient alone is not sufficient: The gradient fails to capture such a perturbation under a fixed perturbation budget, since the same additive perturbation causes a larger relative change to the output for a small-magnitude parameter than for a large-magnitude parameter. Similarly, Kwon et al. [8] observe that the maximum loss change under a fixed-radius additive perturbation is not invariant to re-scaling of the parameters, even when the re-scaling leaves the loss unchanged.

  • Why the weight alone is not sufficient: Molchanov et al. [9] observe that weight magnitude correlates poorly with the change in loss caused by removing a neuron, and hence is a poor indicator of importance.

  • Why masking causes degradation: Model degradation due to masking of WAG-identified parameters is further motivated by Theorem 3.3 together with the fact that LLMs are scale-balanced systems: their parameter magnitudes vary substantially across layers, attention blocks, and MLP blocks [5], yet the models behave stably because the loss depends on the relative balance of these scales rather than on raw parameter values [6, 7]. Optimization therefore settles into a coordinated balance of parameter scales. Theorem 3.2 shows that WAG measures this loss sensitivity to relative parameter scale, and this captures the scale balance. By Theorem 3.3, the WAG vector defines the direction along which this balance is most fragile, and parameters with large magnitude of WAGi\mathrm{WAG}_i, irrespective of sign, dominate this direction. These parameters therefore account for a disproportionate fraction of the model's local scale stability. Masking them removes the coordinates that contribute most to preserving the learned balance of scales, so the model collapses when the parameters along these dominant directions are masked.

We have also made this theoretical explanation clearer in Appendix B.3.

AuthorsOpenReview ↗

Response to Reviewer C2qy [2/3]

W2. Only limited and classical metrics used to estimate parameter importance are compared in experiments. A lot of recent works are not listed, like integrated gradients methods[1], OBD [2], and so on.

[1] Axiomatic Attribution for Deep Networks, ICML 2017
[2] Optimal Brain Damage, NeurIPS 2026

Response: Comparison with OBD and Integrated Gradients

We thank the reviewer for suggesting these methods. We have added OBD [1] as a baseline for identifying the parameters whose masking causes model collapse, and masking WAG-identified parameters causes faster model collapse than masking OBD-identified parameters. We also compare WAG with the recent Model Shapley [3] in our response to Q3. Integrated Gradients [4], on the other hand, attributes predictions to input features and is not a parameter saliency measure:

  • OBD formulation: OBD [1] estimates the loss change caused by removing a parameter with a second-order Taylor expansion. It ignores the first-order term and keeps only the diagonal of the Hessian HH: si=12Hiiθi2s_i = \frac{1}{2} H_{ii} \theta_i^2. For LLMs, we use the LLM-Pruner [2] implementation of this score (its element-wise second-order Taylor importance), which approximates HiiH_{ii} by the empirical Fisher information: si=θi2,E[(∂L∂θi)2]s_i = \theta_i^2 \\, \mathbb{E}\big[(\frac{\partial L}{\partial \theta_i})^2\big]. Hence, OBD excludes the first-order weight–gradient term that WAG measures, and it is designed to find the parameters whose removal changes the loss the least.
  • OBD results: Since OBD selects parameters by a direction-based saliency, we compare it with the direction-based WAG sensitivities, WAGHigh\text{WAG}_\text{High} and WAGLow\text{WAG}_\text{Low}. On Gemma3-12B fine-tuned on coding, we mask the parameters with the highest OBD scores and with the highest (WAGHigh\text{WAG}_\text{High}) and lowest (WAGLow\text{WAG}_\text{Low}) WAG scores, using the same number of masked parameters, and report pass@1 on HumanEval and HumanEval+ (mean ± SEM over 3 seeds). Masking parameters identified by both WAGHigh\text{WAG}_\text{High} and WAGLow\text{WAG}_\text{Low} causes faster model collapse than masking parameters identified by OBD, at every budget on both datasets.

Pass@1 (%) on Gemma3-12B vs. number of parameters masked: WAG vs. OBD

DatasetMethod3,0006,0009,00012,00015,00018,00021,00024,000Average
HumanEvalOBD42.07±6.5835.16±7.8028.25±8.2027.44±7.9624.59±6.6320.33±7.2718.09±9.5816.26±8.1226.52±7.70
WAGLow\text{WAG}_\text{Low}34.35±1.2428.46±2.3421.54±3.0817.07±3.7313.01±3.5410.77±3.387.11±3.357.11±3.4217.43±2.87
WAGHigh\text{WAG}_\text{High}27.44±1.8311.79±0.814.47±1.082.03±0.411.22±0.610.81±0.540.20±0.200.20±0.206.02±0.46
HumanEval+OBD36.38±7.4031.10±7.1924.59±7.8723.78±7.6720.12±6.1017.07±6.7414.63±8.5413.62±7.3222.66±7.21
WAGLow\text{WAG}_\text{Low}29.67±1.4725.20±2.4719.72±2.7314.02±3.1310.77±2.827.52±1.595.69±2.825.49±2.6614.76±2.38
WAGHigh\text{WAG}_\text{High}24.19±1.7410.37±1.413.46±0.892.85±0.890.81±0.540.81±0.410.20±0.200.00±0.005.34±0.55
  • Integrated Gradients: Integrated Gradients [4] studies "the problem of attributing the prediction of a deep network to its input features". For an input xx and a baseline input x′x' (e.g., a black image), it integrates the gradient of the network output FF with respect to the input along the straight path from x′x' to xx: IGi(x)=(xi−xi′)∫01∂F(x′+α(x−x′))∂xi,dα\mathrm{IG}_i(x) = (x_i - x'_i) \int_{0}^{1} \frac{\partial F(x' + \alpha (x - x'))}{\partial x_i} \\, d\alpha. Hence, Integrated Gradients measures the importance of the input data features for a prediction, not the importance of the model parameters. In contrast, WAG identifies the loss-sensitive parameters of the model and does not attribute the prediction to the input data. Therefore, Integrated Gradients is not a parameter saliency measure that can be compared with WAG in our setting.

Q1. Why do you choose to using multiplication to combine the parameter magnitude and the first-order gradient. Have you tried other combination methods such as add or others?

Response: Choice of Multiplication

We explain our motivation for combining the parameter magnitude and the first-order gradient through multiplication in our response to W1 above.

AuthorsOpenReview ↗

Response to Reviewer C2qy [3/3]

Q2. What's the difference between WAG and AdaLoRA? The calculation formulas for these two parameter importance estimation methods look the same.

Response: WAG vs. AdaLoRA

AdaLoRA doesn't use the WAGi=−,θi,∂ℓ(z;θ)∂θi\mathrm{WAG}_i = -\\,\theta_i \\, \frac{\partial \ell(z;\theta)}{\partial \theta_i} term directly; instead, it is only one component of a three-term combined saliency measure, and AdaLoRA does not observe the impact of WAG or its directional components directly. Moreover, the applicability of their saliency remains constrained to one setting, whereas we show that WAG can be extended reliably to multiple settings.

AdaLoRA defines its saliency measure in Equation 7 as Sk,i=s(λk,i)+1d1∑j=1d1s(Pk,ji)+1d2∑j=1d2s(Qk,ij)S_{k,i} = s(\lambda_{k,i}) + \frac{1}{d_1}\sum_{j=1}^{d_1} s(P_{k,ji}) + \frac{1}{d_2}\sum_{j=1}^{d_2} s(Q_{k,ij}), where the first term is given as (θ,∂ℓ∂θ)2⋅U\left(\theta \\, \frac{\partial \ell}{\partial \theta}\right)^2 \cdot U. Hence, it doesn't observe the impact of −,θi,∂ℓ(z;θ)∂θi-\\,\theta_i \\, \frac{\partial \ell(z;\theta)}{\partial \theta_i} and ignores its directional effect. Moreover, AdaLoRA only uses a version smoothed over multiple training steps, whereas WAG is designed to be applied without smoothing, following Theorems 3.1 to 3.3 and our response to W1.



Q3. Will integrating the second-order gradient information bring more benefits to WAG? There are already many good ways to approximate second-order gradient during parameter importance estimation for modern LLMs such as Model Shapley. I think high computational cost or approximation error should not be the obstacle.

Response: Second-order Information and Model Shapley

For identifying the parameters that cause model collapse, integrating second-order information does not bring further benefit to WAG. In our response to W2, we compare WAG with OBD [1], a second-order saliency measure, and show that masking WAG-identified parameters causes faster model collapse. Below, we further compare WAG with Model Shapley [3].

Model Shapley [3] estimates the Shapley value of each parameter with a second-order Taylor expansion of the utility U=−LU = -L: ϕi=−θi∂L∂θi−12θi∑jwijHijθj\phi_i = -\theta_i \frac{\partial L}{\partial \theta_i} - \frac{1}{2} \theta_i \sum_{j} w_{ij} H_{ij} \theta_j, where HH is approximated with the blockwise Fisher information and wijw_{ij} are path weights. Model Shapley selects the parameters with the highest Shapley value. Hence, its formulation is closer to WAGHigh\text{WAG}_\text{High}: selecting by its first term alone gives exactly the WAGHigh\text{WAG}_\text{High} selection, and Model Shapley further introduces a second-order term that captures the cooperative interactions between parameters.

In the same setting as in W2, we compare Model Shapley with WAGHigh\text{WAG}_\text{High}. Although Model Shapley adds the higher-order cooperative term to the WAGHigh\text{WAG}_\text{High} term, the higher-order term does not help identify the parameters that cause model collapse: masking parameters identified by WAGHigh\text{WAG}_\text{High} still causes faster model collapse at every budget on both datasets.

Pass@1 (%) on Gemma3-12B vs. number of parameters masked: WAGHigh\text{WAG}_\text{High} vs. Model Shapley

DatasetMethod3,0006,0009,00012,00015,00018,00021,00024,000
HumanEvalModel Shapley34.96±6.8822.97±9.9019.72±8.4413.62±8.4512.40±8.758.74±7.536.30±5.395.49±4.89
WAGHigh\text{WAG}_\text{High}27.44±1.8311.79±0.814.47±1.082.03±0.411.22±0.610.81±0.540.20±0.200.20±0.20
HumanEval+Model Shapley32.11±6.9921.75±8.3418.09±7.7311.99±7.1210.98±7.947.72±6.505.69±5.085.08±4.78
WAGHigh\text{WAG}_\text{High}24.19±1.7410.37±1.413.46±0.892.85±0.890.81±0.540.81±0.410.20±0.200.00±0.00

References:

[1] Optimal Brain Damage, NeurIPS 1989
[2] LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS 2023
[3] MODEL SHAPLEY: Find Your Ideal Parameter Player via One Gradient Backpropagation, NeurIPS 2025
[4] Axiomatic Attribution for Deep Networks, ICML 2017
[5] Systematic Outliers in Large Language Models, ICLR 2025
[6] Sharp Minima Can Generalize For Deep Nets, ICML 2017
[7] Layer Normalization, ArXiv 2016
[8] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks, ICML 2021
[9] Importance Estimation for Neural Network Pruning, CVPR 2019

Reviewer_aNghOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

(All my reviews are rewritten)

This paper defines Weight-Adjusted Gradients, WAG, as wi=−θi∂ℓ/∂θiw_i = -\theta_i \partial\ell/\partial\theta_i, and uses it to score each parameter. Theorems 3.1 to 3.3 explain it as loss sensitivity to small multiplicative changes. Masking up to 2 ppm of WAG-selected parameters makes three LLMs collapse. The paper also uses WAG for LoRA expert allocation, unlearning, quantization and knowledge editing.

Critical strengths

S1. The masking study has its own baselines. Figures 1 and 5 also show weight and gradient selection at the same budgets. On Gemma3-12B, the WAG curves are clearly apart. I think this is the strongest evidence in the paper.

S2. The score is cheap. It needs only the gradient and one product, with no retraining.

Critical weaknesses

W1. The score is not new, and the theory does not explain the collapse.

May main concerns include: 1) In Appendix A, Molchanov et al. (2017) already use this first-order Taylor term for parameter importance. Still, the paper calls the relation unexplored; 2) As forTheorem 3.2, it needs small ϵi\epsilon_i, but masking to zero gives ϵi=−1\epsilon_i = -1. The first-order change ∑i∈Swi\sum_{i \in S} w_i says masking WAG-Low should lower the loss; 3) Following by this the Figure 1, WAG-Low collapses faster than WAG-High.

W2. The paper's own figures and tables do not support the main empirical claims.

May main concerns include: 1) Is the effect specific to WAG? On LLaMA3.2-3B, |Weight| also falls to near zero at the first budget, just like |WAG| in Figure 5; 2) Which data is used for scoring? Appendix E.1 computes gradients on the evaluation set. Table 3 scores quantization groups on the evaluation set and reports HumanEval. 3) The gains in Table 1 are small. The best LLaMA3.2-3B gain is 61.87±2.049 vs 60.77±0.364. 4) Some baselines are missing. Table 2 has no random, weight or gradient choice of 75% of layers. Table 4 compares only with CMA, and Table 1 has no LayerIF.

Questions for authors

Questions

Q1. What does WAG add beyond Taylor saliency, such as Molchanov et al. 2017 and Lee et al. 2019?

Q2. Are the scoring gradients computed on the samples you evaluate? Please rerun Table 3 with data that does not overlap with HumanEval.

Q3. On LLaMA3.2-3B, how much do the |WAG| and |Weight| selections overlap? What does random masking do?

Q4. What does mask mean? Does the measured loss change match ∑i∈Swi\sum_{i \in S} w_i?

Q5. Do you score and mask the base model or the fine-tuned one? Please give the training details.

Suggestions:

Suggestion 1 The paper tests a fixed grid of budgets. It shows a number that is enough, not the minimum.

Suggestion 2 Three sentences say consistently or every, but Tables 1, 2 and 4 each have a small exception.

Suggestion 3* 6,665 out of 224M parameters is about 0.003%, not 0.0003%.

References to correct (quoted exactly as printed in the submission):

  1. "Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. SliM-LLM: Salience-driven mixed-precision quantization for large language models. In International Conference on Machine Learning (ICML), 2025." The ICML 2025 version has ten authors. Qinshuo Liu (fifth) and Shiming Zhang (ninth) are missing.
  2. "Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. PB-LLM: Partially binarized large language models. In International Conference on Learning Representations (ICLR), 2024." The ICLR 2024 version has three authors: Zhihang Yuan, Yuzhang Shang and Zhen Dong. Qiang Wu is not an author, and the first two names are in the wrong order.

These are examples, not a full list. Please check the whole bibliography.

7 comments
AuthorsOpenReview ↗

Response to Reviewer aNgh [1/3]

Dear Reviewer aNgh,
Thank you for your detailed review. We provide clarifications below. In tables, the best is bold and the second best is highlighted.


W1.1. Section 3.1 … "unexplored".

Response: Novelty of WAG over Taylor-based Saliency

We would like to respectfully clarify that the first-order Taylor approximations in prior works (Appendix A, lines 151 to 153) do not derive the WAG score, and the loss sensitivity to multiplicative perturbation that WAG captures remains unexplored:

  • OBD [2] omits the first-order weight–gradient relation and uses only the second-order Taylor term.
  • Lee et al. [3] use ∂L∂ci\frac{\partial L}{\partial c_i} for pruning, where ci∈{0,1}c_i \in \{0,1\} indicates whether the weight is connected.
  • Molchanov et al. [1] treat LL as a function of the activation hih_i, with removal setting hi=0h_i = 0, so their first-order term is hi∂L∂hih_i \frac{\partial L}{\partial h_i}.
  • WAG, −θi∂L∂θi-\theta_i \frac{\partial L}{\partial \theta_i}, arises from the multiplicative perturbation θi′=θi(1+ϵi)\theta_i' = \theta_i(1+\epsilon_i); by Theorem 3.2, ℓ(z;θ′)−ℓ(z;θ)=−∑iϵi WAGi+O(∥ϵ∥2),\ell(z;\theta') - \ell(z;\theta) = -\sum_{i} \epsilon_i \, \mathrm{WAG}_i + O(\|\epsilon\|^2), so WAG directly measures each parameter's loss sensitivity to relative perturbation.

Therefore, none of these works explores the WAG's relation between weight and gradient.



W1.2. Theorem 3.2 needs small ϵi\epsilon_i … degradation".

Response: Theoretical Link to Model Collapse

We would like to respectfully clarify that we use Theorem 3.2 with small ϵi\epsilon_i to relate WAG to loss sensitivity, not to predict the loss change under masking (setting a weight to 0), which is indeed not a small perturbation. The theoretical reason why masking WAG-identified parameters induces model degradation comes from Theorem 3.3 and Appendix B.3:

  • WAG as loss sensitivity: By Theorem 3.2 (equation in W1.1), WAG is each parameter's loss sensitivity to small multiplicative perturbation, so it identifies the parameters to which the loss is most sensitive.
  • Masking: Masking sets these parameters to 0, i.e., the same multiplicative perturbation of −1 for each.
  • Why masking causes collapse: LLMs are scale-balanced systems [7, 8, 9]. Theorem 3.2 shows that WAG measures the loss sensitivity to relative parameter scale, and this captures the scale balance. By Theorem 3.3, the WAG vector defines the direction along which this balance is most fragile, and parameters with large magnitude of WAGi\mathrm{WAG}_i, irrespective of sign, i.e., in either direction of WAGLow\text{WAG}_\text{Low} or WAGHigh\text{WAG}_\text{High}, dominate this direction. These parameters therefore account for a disproportionate fraction of the model's local scale stability. Masking them removes the coordinates that contribute most to preserving the learned balance of scales, so the model collapses when the parameters along these dominant directions are masked.

We have clarified this explanation in Appendix B.3.



W2.1. Is the effect specific to WAG? … show a gap.

Response: Specificity of Model Collapse to WAG

We would like to respectfully clarify that the collapse is specific to WAG: the cases where masking Weight-identified parameters collapses the model as much as WAG occur only on LLaMA3.2-3B:

  • Figure 1: Masking Weight-identified parameters causes a collapse similar to WAG in only 2 of 12 settings (3 models × 4 datasets): LLaMA3.2 on HumanEval and HumanEval+.
  • Figure 5: Masking Weight-identified parameters causes a collapse similar to WAG in only 4 of 12 settings: LLaMA3.2 on HumanEval, HumanEval+, GSM8K, and HellaSwag.
  • Overall: The collapse is similar in only 6 of 24 settings, all on LLaMA3.2. In the other 18, masking WAG-identified parameters causes a stronger collapse at the same budget.


W2.2. Which data is used for scoring? …

Response: Held-out Data for Scoring

We have revised the paper to compute WAG scores on a 5% held-out training set, disjoint from the evaluation data, and the trends of the updated results remain the same:

  • Empirical analysis: Masking WAG-identified parameters causes similar or faster collapse than masking gradient- or weight-identified parameters.
  • Applications A and C: WAG-based methods remain in the top 2 for MoE expert allocation and outperform other baselines on average for mixed-precision quantization.
  • Applications B and D: Unchanged, as unlearning scores on the forget samples and knowledge editing on a proxy set disjoint from the test set, both available beforehand.
  • OBS: Calibrated on the same 5% held-out set, or on WikiText-2 (the standard data in [4]), OBS performs similarly to or worse than WAG:
ModelOBS (5% held-out)OBS (WikiText-2)WAG (5% held-out)OBS+|WAG| (5% held-out)
LLaMA3.2-3B33.38±1.08134.76±0.53534.76±0.40335.42±0.356
Gemma3-12B52.44±0.76752.44±1.50452.79±0.68453.61±0.500
AuthorsOpenReview ↗

Response to Reviewer aNgh [2/3]

W2.3. The gains in Table 1 are small. …

Response: Statistical Significance of the Gains

We would like to emphasize that small gains are typical for MoE expert allocation, and the gain of the best WAG variant over the best baseline is statistically significant for both models:

  • Small gains are typical: LayerIF [6] reports that its best configuration improves over the best baseline by only 0.34 points on Gemma-7B on average.
  • Test design: For simplicity, we consider the hypothesis to be the best WAG variant vs. the best baseline (table below), with a one-sided paired t-test. Each pair is one test sample scored by both methods, with its accuracy averaged over 3 seeds, pooled across MRPC, CoLA, and ScienceQA (4,992 pairs per model).
ModelNull Hypothesisp-value
LLaMA3.2-3B|WAG| is not better than MoLA Hourglass3.5×10⁻⁷ < 0.05
Gemma3-12BWAGLow\text{WAG}_\text{Low} is not better than MoLA Rectangle2.6×10⁻³ < 0.05


W2.4. Some baselines are missing. …

Response: Additional Baselines

We would like to emphasize that our goal is to show the diverse applicability of WAG rather than to beat application-specific approaches; nonetheless, with all suggested baselines added, WAG remains among the top approaches:

A. Targeted Unlearning (Table 2). From the table below, |WAG| selects better layers than random, weight-, and gradient-based selection, and the three WAG-based approaches are among the top 4 for both metrics. Thus, the gains of |WAG| come from which layers it selects, whereas the Figure 4 speedup comes from updating fewer layers.

GradDiff layersForget TRKS Forget
All layers0.497±0.0560.588±0.014
Random0.475±0.0410.583±0.008
Magnitude0.517±0.0310.571±0.009
Gradient0.578±0.0540.596±0.028
WAGLow\text{WAG}_\text{Low}0.546±0.0430.607±0.015
WAGHigh\text{WAG}_\text{High}0.541±0.0230.599±0.013
|WAG|0.589±0.0060.613±0.007

B. Knowledge Editing (Table 4). For R-ROME editing on ZSRE, |WAG| is among the top 2 on both models (best Overall on Gemma3-12B, second on LLaMA3.2-3B, close behind LGA), even though it is not designed for KE, and needs one gradient computation instead of two for LGA.

ModelSelectionRwARpALOCPRTOV
LLaMA3.2-3BCMA0.99460.95950.94200.53830.8586
SaLEM0.96090.91740.93480.53680.8375
LGA0.99400.97120.95220.53880.8641
|WAG|0.99060.94830.96240.54330.8612
Gemma3-12BCMA0.85810.70410.94210.49890.7508
SaLEM0.95750.81500.96460.50280.8100
LGA0.95350.87960.77360.51980.7816
|WAG|0.95750.81500.96460.50280.8100

C. MoE Expert Allocation (Table 1). WAG outperforms LayerIF on average on Gemma3-12B, both scored on the same 5% held-out training set:

MethodMRPCCoLAScienceQAAverage
MoLA Rectangle73.41±2.3583.41±0.4293.18±0.2183.33±0.84
LayerIF72.17±2.4883.38±0.2592.10±0.2082.55±0.80
WAGLow\text{WAG}_\text{Low}75.94±0.4482.04±0.1393.60±0.0483.86±0.20
WAGHigh\text{WAG}_\text{High}76.06±1.7782.61±0.1992.19±0.1783.62±0.61
|WAG|72.25±3.5783.89±0.6293.21±0.2983.12±1.30

Q1. What does WAG add beyond … ?

Response: WAG vs. First-order Taylor Saliency

Beyond the first-order Taylor saliency cited in Section 3.1, WAG studies the unexplored weight–gradient relation expressing loss sensitivity to multiplicative perturbation, and |WAG| selects the highest-saliency parameters, whereas Molchanov et al. [1] and Lee et al. [3] select the lowest. Details are in our responses to W1.1 and W1.2.



Q2. Are the scoring gradients … HumanEval.

Response: No Overlap between Scoring and Evaluation Data

In the revised paper, the scoring gradients are not computed on the evaluated samples, as explained in our response to W2.2. For Table 3, we sample two mutually exclusive sets from Evol CodeAlpaca, CodeFeedback, and Tulu3 Persona Python (released by MFT [5]): about 1,500 samples for WAG scoring and about 30,000 for training. Hence, the scoring data does not overlap with HumanEval, which is separate from both.



Q3. On LLaMA3.2-3B, how much … same budget?

Response: Overlap and Random Masking

The |WAG| and |Weight| selections overlap very little, and random masking at the same budget does not affect performance, while masking |WAG|- or |Weight|-selected parameters leads to collapse. On LLaMA3.2-3B (coding), we mask 0.5 parts per million (ppm) of the parameters, i.e., 1,409, selected by |WAG|, |Weight|, or at random from the same set. Only about 2% of the |WAG|-selected parameters are also selected by |Weight|: 29 (2.06%), 28 (1.99%), and 28 (1.99%) for seeds 42, 43, and 44.

SelectionHumanEvalHumanEval+
Random39.84±0.4135.98±0.93
|Weight|0.00±0.000.00±0.00
|WAG|0.00±0.000.00±0.00
AuthorsOpenReview ↗

Response to Reviewer aNgh [3/3]

Q4. What does "mask" mean? … match ∑i∈Swi\sum_{i \in S} w_i?

Response: Masking and Loss Change

By masking, we mean setting each selected weight to zero, θi′=0\theta_i' = 0 for i∈Si \in S, with all other weights unchanged. The measured loss change matches ∑i∈SWAGi\sum_{i \in S} \mathrm{WAG}_i in sign, and its magnitude mostly tends to increase with the number of masked weights, as shown below on LLaMA3.2-3B fine-tuned on GSM8K (WAG and the loss before and after masking computed on the 5% held-out training set):

Weights maskedWAGHigh\text{WAG}_\text{High}: ΣWAGWAGHigh\text{WAG}_\text{High}: ΔLWAGLow\text{WAG}_\text{Low}: ΣWAGWAGLow\text{WAG}_\text{Low}: ΔL
1+0.012+0.009−0.005−0.004
10+0.043+8.65−0.029−0.029
50+0.081+8.28−0.075−0.063
100+0.115+8.62−0.113−0.080
200+0.168+8.60−0.175−0.094


Q5. Which checkpoint … training details.

Response: Checkpoints and Fine-tuning Details

All scoring and masking in Section 3 and Appendix E use the fine-tuned checkpoint of each task. The Appendix F sentence referred to the base checkpoints, initialized from the original (unquantized) pretrained HuggingFace weights (meta-llama/Llama-3.2-3B, Qwen/Qwen3-4B, google/gemma-3-12b-pt); we revised it to: "All the base LLMs are original (unquantized) open-source versions from HuggingFace."

Fine-tuning details (full-parameter fine-tuning, bf16, following MFT [5]):

CodingGSM8K / HellaSwag
Training data30,000 (10,000 each from Tulu3 Personas Code, Evol CodeAlpaca, Codefeedback)GSM8K train / HellaSwag train
Epochs23
Learning rate / schedule2e-5, linear, 3% warmupsame
Weight decay00
Per-device batch × grad. accum.1 × 161 × 16
GPUs (effective batch)LLaMA 2 (32), Gemma 4 (64)same
Max sequence length4,096LLaMA 4,096 / Gemma 2,048


Q6. Lines 122 to 123 … not the minimum.

Response: Minimum Number of Weights

We agree. We do not solve for the minimum as an optimization problem; our grid search shows that masking as few as 0.5 ppm of the parameters, identified by WAG, induces collapse. We revised lines 122–123 to:

"However, these works do not identify the critical parameters or how few weights are sufficient to induce model collapse."



Q7. Three sentences … but CMA has 0.5796.

Response: Revised Claims

We apologize for the lack of clarity.

(1) Table 1: Our sentence explicitly states "on average", which matches our results: the best WAG-guided allocation has a higher average accuracy than all baselines on both models (|WAG| over MoLA Hourglass on LLaMA3.2-3B, WAGLow\text{WAG}_\text{Low} over MoLA Rectangle on Gemma3-12B).

(2) Table 2: WAG-targeted GradDiff improves over GradDiff in 17 of 18 cases (3 criteria × 3 metrics × 2 models); we now state the exception explicitly: "All WAG criteria improve all three metrics, except |WAG| utility on LLaMA3.2-3B."

(3) Table 4: This is our mistake: the claim refers to the Overall score, not to Locality, where |WAG| matches or exceeds CMA in all cases except Gemma3-12B WikiRecent. We separated the two statements: "The gains are most pronounced for Locality and Portability. Overall, ∣WAG∣|\text{WAG}| matches or exceeds CMA in every setting except for WikiBio on LLaMA3.2."



Q8. Line 314: … not 0.0003%.

Response: Typo Correction

We thank the reviewer for catching this typo; it is about 0.003%. With WAG scored on the 5% held-out training set, the updated paper reads: "At most, 4,234 parameters ... at most 0.002% of a layer is masked."



Q9. Please add a limitations section. …

Response: Limitations and Ethics Statement

We agree and have added a Limitations section and revised the Ethics Statement: although masking a tiny set of WAG-identified parameters can collapse an LLM, only the developers can modify their model's parameters, and they could already collapse it by masking many parameters; hence, WAG introduces no new risk.



Q10. A. "Wei Huang, … are missing.

B. "Yuzhang Shang, … wrong order.

Response: Reference Corrections

We apologize for this oversight. This is an unusual case, as both author lists changed between the original arXiv versions and the conference versions. We followed the original arXiv author lists, and when updating the venues to ICML 2025 and ICLR 2024 we missed the updated author lists.

Original arXiv versions: (A) https://arxiv.org/abs/2405.14917v1, (B) https://arxiv.org/abs/2310.00034. We have corrected both entries to match the published versions.


References:

[1] Pruning CNNs for Resource Efficient Inference, ICLR 2017
[2] Optimal Brain Damage, NeurIPS 1989
[3] SNIP, ICLR 2019
[4] SliM-LLM, ICML 2025
[5] Mask Fine-Tuning (MFT), ArXiv 2025
[6] LayerIF, NeurIPS 2025
[7] Systematic Outliers in Large Language Models, ICLR 2025
[8] Sharp Minima Can Generalize For Deep Nets, ICML 2017
[9] Layer Normalization, ArXiv 2016

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Paper PDF Update

Dear Reviewers and Area Chair,
We thank all reviewers for their thoughtful feedback and the effort spent on our work. We have uploaded a revised PDF, and we summarize the main changes below.


Summary of Revisions

  • Held-out scoring: WAG scores and the baselines are now computed on a 5% held-out training set, which is not seen during evaluation and does not overlap with the evaluation sets. The performance trends remain the same, and the corresponding results are updated:
    • Empirical analysis (Section 3 and Appendix D): Figures 1, 3, 5, 8, and 9.
    • Application A, MoE expert allocation (Section 4): Table 1.
    • Application C, mixed-precision quantization (Section 4): Table 3.
    • The results of Application B (Table 2) and Application D (Table 4) remain unchanged, as their scores are computed on the forget samples and the proxy set, respectively.
  • Additional baselines (Appendix F): We add random masking, OBD, and Model Shapley for the empirical analysis (Tables 5 to 7); LayerIF for MoE expert allocation (Table 8); random, magnitude-based, and gradient-based layer selection for targeted unlearning (Table 9); and SaLEM and LGA for knowledge editing (Table 10). The overall performance trend remains similar, where WAG mostly remains the top performing approach.
  • Limitations and Ethics Statement: We add a Limitations section in Appendix G and revise the Ethics Statement to clarify that WAG does not introduce a new risk, as only model developers can modify their model's parameters.
  • Revised claims and corrections: We revise the statements on the minimum number of masked weights (Introduction), the Table 2 and Table 4 claims (Section 4), and the percentage of masked parameters (Section 3); clarify the base models (Appendix H); and update the SliM-LLM and PB-LLM author lists (References).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.