- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Global Relative Kinetic Utility (Global RKU), a label-free method for global structured pruning of large language models. The method estimates FFN channel importance using activation–gradient interactions with the final hidden-state representation, followed by block-relative normalization to enable cross-layer comparison under a global sparsity budget. Experiments on Qwen-2.5-7B, Llama-3.1-8B, and Gemma-2-9B evaluate pruning accuracy, calibration sensitivity, post-pruning recovery, and inference efficiency.
Critical strengths
-
The paper identifies a relevant challenge in global structured pruning: importance scores from different layers may not be directly comparable under a shared sparsity budget. Separating channel importance estimation from cross-layer calibration provides a clear motivation for the proposed method.
-
The controlled normalization ablation shows substantial improvements, with Mean3 increasing from 17.94 to 32.36 at 40% sparsity and from 16.80 to 22.33 at 50% sparsity (Table 6). The experiments also cover multiple model families, structural seeds, reasoning tasks, and post-pruning recovery.
-
The method requires no downstream labels or iterative pruning optimization. The physical deployment results are also useful, showing practical memory and inference improvements over the dense model.
Critical weaknesses
-
The proposed importance criterion lacks sufficient evidence of its functional relevance. RKU estimates channel importance using the final hidden-state norm, yet it remains unclear how reliably this signal reflects actual performance degradation after channel removal. While Table 6 shows improvements of 3.96/1.30 Mean3 points over the LM-CE probe at 40/50% sparsity, the paper does not directly validate whether the proposed scores identify channels that are genuinely important for preserving model capabilities.
-
The main performance gains are not sufficiently isolated from the normalization strategy. Table 6 shows that relative normalization alone contributes 14.42/5.53 Mean3 points at 40/50% sparsity, substantially more than the improvement from the proposed importance probe. Since the Wanda and Taylor baselines use fixed layer-wise quotas, the paper lacks a matched comparison applying the same global normalization and selection strategy to existing importance scores, leaving the specific contribution of RKU unclear.
-
The improvements over competitive global pruning methods do not generalize consistently across architectures. Under matched Q16K calibration, RKU outperforms GISP on Qwen by 3.79/8.61 Mean3 points at 40/50% sparsity, but underperforms on Llama by 3.91/1.92/1.63 points and Gemma by approximately 3.50/3.31/3.18 points at 30/40/50% sparsity (Appendix Tables 14, 18, 19, and 21). As the two methods also use different pruning spaces, the current evidence is insufficient to establish a general advantage of the proposed scoring and allocation mechanism.
Questions for authors
-
How well do the proposed importance scores correlate with actual model degradation when individual channels are removed?
-
Would an existing importance criterion, such as Taylor-based channel saliency, achieve comparable performance when equipped with the same global normalization and selection procedure as RKU?
-
What explains RKU's consistently weaker results against GISP on Llama and Gemma under matched calibration, and does this difference persist when both methods use the same FFN-only pruning space?
2 comments
A Gentle Reminder of Further Feedback — Reviewer Fe56
Dear Reviewer Fe56,
As the author–reviewer discussion period approaches its conclusion, we would greatly appreciate any further feedback on our detailed rebuttal and revised manuscript.
We have carefully addressed your concerns by adding direct single-channel deletion experiments, comparisons with existing importance scores under the same normalization and global selection procedure, and matched FFN-only evaluations across model families. We have also explicitly acknowledged the settings in which RKU does not outperform GISP.
We hope these additional experiments and clarifications address your principal concerns. We would be grateful to know whether any important questions remain or whether the new evidence changes your assessment.
Thank you again for your time and constructive suggestions.
Best regards,
Authors of Paper 12393
Response to Reviewer Fe56
W1 / Q1. Single-channel deletion. RKU scores distinguish deletion damage at the high and low ends of the ranking within the same FFN block. We start each deletion from the unpruned checkpoint, independently zero each of the three highest- and three lowest-scored channels per block, and measure the change in correct-answer NLL on the same 36 held-out items. Deleting higher-scored channels raises NLL by 0.0217 more on Gemma and 0.0151 more on Llama. Both 95% block-bootstrap intervals are above zero (Table A). This directly tests score separation within a block. The joint pruning test is in Table D.
W2 / Q2. Probe effect and alternative scores. Relative normalization contributes the larger gain, and the final-state probe adds 3.96/1.30 Mean3 points on Qwen at 40/50% with normalization and selection fixed (Table D). The normalization contrast adds 14.42/5.53. We also apply the same relative normalization and global FFN selector to Wanda and Taylor-CE. Taylor-CE uses cross-entropy saliency with global selection. The original Taylor-FO baseline uses fixed layer quotas. Pooled Qwen sensitivity margins are +2.48/+4.29 points. Table C reports the separate Gemma Q16K full-test comparison.
W3 / Q3. Does the ranking persist with FFN-only pruning? On Qwen, Global-RKU wins all 9/9 Full16K pairs with matched text, 16K-token budget, FFN space, and retained parameters. The margins at 30/40/50% are +0.30/+3.04/+3.12 points. GISP’s means remain higher on Llama and Gemma when attention is fixed and both methods prune FFN channels at common budgets (Table E).
A. Within-block single-channel deletion, mean ΔNLL
B. Qwen score controls, sensitivity of Mean3 gaps (pp)
The sensitivity calculation adds 2.20 and 4.90 points to the RKU means in bundles 2 and 3. Bundles are equally weighted, with nine pairs each.
C. Gemma Q16K full-test 3-shot score controls, Mean3 gaps (pp)
D. Qwen probe and normalization ablation, Mean3 (%)
E. Common FFN budgets, three-seed Mean3 (%) at 30/40/50%