- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a Weight-Adjusted Gradients method which utilizes zero-order and first-order information to identify the important weights in the LLM. The authors provide a series of downstream task to show the application and effectiveness of the Weight-Adjusted Gradients method.
Critical strengths
- The paper is clear and easy to follow.
- The proposed method is concise.
- The experiment results are comprehensive and the findings about the impact of important weights in LLM is insightful.
Critical weaknesses
-
The Weight-Adjusted Gradients methodis mathematically similar to the existing Taylor-based saliency measure, which limits the novelty.
-
The improvement on MoE expert allocation is limited and not statistically significant, also for kjnowledge editing.
-
The targeted unlearning experiments do not compare WAG-based layer selection with random layer selection.
Questions for authors
-
Could the authors add some baselines such as random layer selection in targeted unlearning experiments?
-
Could the authors provide more statistically significant experiment results on MoE expert allocation and knowledge editing?
5 comments
Response to Reviewer eP7R [2/3]
W2. The improvement on MoE expert allocation is limited and not statistically significant, also for kjnowledge editing.
Response: Statistical Significance on MoE and Knowledge Editing
We thank the reviewer for raising this concern about the statistical significance of the improvements on MoE expert allocation and knowledge editing. We show below that the gains of WAG are statistically significant. On MoE, although the gains appear to be small, the gain of the best WAG variant over the best baseline is statistically significant for both models. On knowledge editing, |WAG| performs statistically better than CMA across all datasets for Gemma3, and is not statistically worse than CMA in 6 of the 7 tested settings:
- MoE expert allocation: For simplicity, we consider the hypothesis to be the best-performing WAG variant vs. the best baseline. We conduct a one-sided paired t-test between the performance of the best baseline, MoLA Hourglass, and the best WAG variant, |WAG|, for the LLaMA3.2-3B model. For the Gemma3-12B model, the test is conducted between the performance of MoLA Rectangle and . Each pair is one test sample scored by both methods, where the score is that sample's accuracy averaged over the 3 seeds. The samples are pooled across all datasets (MRPC, CoLA, and ScienceQA), giving 4,992 pairs per model. In both cases, the p-value is below 0.05, indicating that the performance gain of the best WAG variant over the best baseline is statistically significant:
One-sided paired t-test: best WAG variant vs. best baseline
| Model | Null Hypothesis | p-value |
|---|---|---|
| LLaMA3.2-3B | |WAG| is not better than MoLA Hourglass | 3.5×10⁻⁷ < 0.05 |
| Gemma3-12B | is not better than MoLA Rectangle | 2.6×10⁻³ < 0.05 |
-
Small gains are typical for MoE expert allocation: In prior work, LayerIF [4] reports that its best configuration improves over the best baseline by 0.34 points on Gemma-7B on average.
-
Knowledge editing: We conduct a similar one-sided sample-wise paired t-test based on the Overall metric. The paired t-test shows that across all datasets for Gemma3, |WAG|'s performance gain over CMA is statistically significant with p<0.05. On the other hand, CMA performs statistically better than |WAG| in only one of the 7 tested settings (WikiBio on LLaMA3.2), covering Gemma3 and LLaMA3.2 on ZSRE, WikiCounterFact, WikiBio, and WikiRecent. On WikiRecent with LLaMA3.2, both methods select the same layer, so no test applies.
One-sided paired t-test, null hypothesis: |WAG| is not better than CMA
| Dataset | Model | p-value |
|---|---|---|
| ZSRE | LLaMA3.2-3B | 1.4×10⁻¹ ≥ 0.05 |
| Gemma3-12B | 5.6×10⁻³⁵ < 0.05 | |
| WikiCounterFact | LLaMA3.2-3B | 2.8×10⁻¹ ≥ 0.05 |
| Gemma3-12B | 8.9×10⁻³⁷ < 0.05 | |
| WikiBio | LLaMA3.2-3B | 9.7×10⁻¹ ≥ 0.05 |
| Gemma3-12B | 2.0×10⁻³⁰ < 0.05 | |
| WikiRecent | LLaMA3.2-3B | n/a (same layer selected) |
| Gemma3-12B | 3.8×10⁻² < 0.05 |
One-sided paired t-test, null hypothesis: CMA is not better than |WAG|
| Dataset | Model | p-value |
|---|---|---|
| ZSRE | LLaMA3.2-3B | 8.6×10⁻¹ ≥ 0.05 |
| Gemma3-12B | 1.0 ≥ 0.05 | |
| WikiCounterFact | LLaMA3.2-3B | 7.2×10⁻¹ ≥ 0.05 |
| Gemma3-12B | 1.0 ≥ 0.05 | |
| WikiBio | LLaMA3.2-3B | 3.0×10⁻² < 0.05 |
| Gemma3-12B | 1.0 ≥ 0.05 | |
| WikiRecent | LLaMA3.2-3B | n/a (same layer selected) |
| Gemma3-12B | 9.6×10⁻¹ ≥ 0.05 |
Response to Reviewer eP7R [3/3]
W3. The targeted unlearning experiments do not compare WAG-based layer selection with random layer selection.
Response: Random Layer Selection Baseline
From the table below, we see that WAG criteria select better layers for targeted unlearning than random selection. The three WAG-based approaches are among the top 3 for targeted unlearning for both metrics.
### Targeted unlearning layer selection: WAG vs. random
| Method | Forget TR | KS Forget |
|---|---|---|
| GradDiff (all layers) | 0.497±0.056 | 0.588±0.014 |
| GradDiff + Random | 0.475±0.041 | 0.583±0.008 |
| GradDiff + | 0.546±0.043 | 0.607±0.015 |
| GradDiff + | 0.541±0.023 | 0.599±0.013 |
| GradDiff + |WAG| | 0.589±0.006 | 0.613±0.007 |
Q1. Could the authors add some baselines such as random layer selection in targeted unlearning experiments?
Response: Random Layer Selection Baseline
WAG-based layer selections outperform random layer selection for the task of targeted unlearning, as shown in our response to W3.
Q2. Could the authors provide more statistically significant experiment results on MoE expert allocation and knowledge editing?
Response: Statistical Significance Results
Statistical significance tests show that |WAG| performs statistically better than CMA across all datasets for Gemma3, and CMA performs statistically better than |WAG| only on the WikiBio dataset for the LLaMA3.2 model. We provide detailed statistical significance results in response to W2.
References:
[1] Pruning Convolutional Neural Networks for Resource Efficient Inference, ICLR 2017
[2] Optimal Brain Damage, NeurIPS 1989
[3] SNIP: Single-shot Network Pruning based on Connection Sensitivity, ICLR 2019
[4] LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions, NeurIPS 2025
Response to Reviewer eP7R [1/3]
Dear Reviewer eP7r,
Thank you for your feedback and for the effort spent on our work. We provide clarifications below. In tables, the best is bold and second best is
highlighted.W1. The Weight-Adjusted Gradients methodis mathematically similar to the existing Taylor-based saliency measure, which limits the novelty.
Response: Novelty over Taylor-based Saliency
Below, we discuss how the Weight-Adjusted Gradients (WAG) method differs from the existing Taylor-based saliency measures mathematically, none of which directly measures or observes the impact of WAG.
Therefore, none of the previous works directly explores the WAG term, and they do not study the direct impact of the most sensitive parameters identified by WAG.