acceptodds
Under review as a conference paper at ICLR 2027

Relative Kinetic Utility: Calibrating Cross-Layer Credit for Global Structured LLM Pruning

Abstract

Global structured pruning requires channels from different layers to compete under a shared sparsity budget, raising two coupled challenges: identifying which channels should be retained and making their scores comparable across layers. Raw channel scores can contain block-common scale that leaves within-block ordering unchanged but distorts model-wide competition. Our experiment indicates that similar layer-wise allocations can retain substantially different FFN channels, so layer allocation alone does not determine channel identity. Motivated by this separation, we introduce Global Relative Kinetic Utility (Global RKU), a label-free criterion that separates channel importance estimation from cross-layer comparison. Global RKU measures channel participation using a final-hidden-state activation–gradient signal, then applies block-relative normalization to mitigate block-common scale while preserving within-block ordering, requires only unlabeled calibration inputs, and produces a static pruning topology in a single calibration stage. Under questions-only calibration on Qwen-2.5-7B, RKU–GISP Mean3 margins are at 30/40/50% sparsity (average ). Additional Qwen evaluations cover non-mathematical reasoning, recovery, held-out transfer, and physical deployment. Separately, replacing Wiki16K with questions-only Q16K improves RKU's Mean3 at every tested sparsity on Qwen, Llama, and Gemma. Our ablation study shows relative-normalization gains of 14.42/5.53 Mean3 points at 40/50% sparsity; the common-seed audit is positive in all 27 seed–task comparisons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model compression, structured pruning, global pruning, importance estimation, label-free calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fe56OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Global Relative Kinetic Utility (Global RKU), a label-free method for global structured pruning of large language models. The method estimates FFN channel importance using activation–gradient interactions with the final hidden-state representation, followed by block-relative normalization to enable cross-layer comparison under a global sparsity budget. Experiments on Qwen-2.5-7B, Llama-3.1-8B, and Gemma-2-9B evaluate pruning accuracy, calibration sensitivity, post-pruning recovery, and inference efficiency.

Critical strengths
  1. The paper identifies a relevant challenge in global structured pruning: importance scores from different layers may not be directly comparable under a shared sparsity budget. Separating channel importance estimation from cross-layer calibration provides a clear motivation for the proposed method.

  2. The controlled normalization ablation shows substantial improvements, with Mean3 increasing from 17.94 to 32.36 at 40% sparsity and from 16.80 to 22.33 at 50% sparsity (Table 6). The experiments also cover multiple model families, structural seeds, reasoning tasks, and post-pruning recovery.

  3. The method requires no downstream labels or iterative pruning optimization. The physical deployment results are also useful, showing practical memory and inference improvements over the dense model.

Critical weaknesses
  1. The proposed importance criterion lacks sufficient evidence of its functional relevance. RKU estimates channel importance using the final hidden-state norm, yet it remains unclear how reliably this signal reflects actual performance degradation after channel removal. While Table 6 shows improvements of 3.96/1.30 Mean3 points over the LM-CE probe at 40/50% sparsity, the paper does not directly validate whether the proposed scores identify channels that are genuinely important for preserving model capabilities.

  2. The main performance gains are not sufficiently isolated from the normalization strategy. Table 6 shows that relative normalization alone contributes 14.42/5.53 Mean3 points at 40/50% sparsity, substantially more than the improvement from the proposed importance probe. Since the Wanda and Taylor baselines use fixed layer-wise quotas, the paper lacks a matched comparison applying the same global normalization and selection strategy to existing importance scores, leaving the specific contribution of RKU unclear.

  3. The improvements over competitive global pruning methods do not generalize consistently across architectures. Under matched Q16K calibration, RKU outperforms GISP on Qwen by 3.79/8.61 Mean3 points at 40/50% sparsity, but underperforms on Llama by 3.91/1.92/1.63 points and Gemma by approximately 3.50/3.31/3.18 points at 30/40/50% sparsity (Appendix Tables 14, 18, 19, and 21). As the two methods also use different pruning spaces, the current evidence is insufficient to establish a general advantage of the proposed scoring and allocation mechanism.

Questions for authors
  1. How well do the proposed importance scores correlate with actual model degradation when individual channels are removed?

  2. Would an existing importance criterion, such as Taylor-based channel saliency, achieve comparable performance when equipped with the same global normalization and selection procedure as RKU?

  3. What explains RKU's consistently weaker results against GISP on Llama and Gemma under matched calibration, and does this difference persist when both methods use the same FFN-only pruning space?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Fe56

W1 / Q1. Single-channel deletion. RKU scores distinguish deletion damage at the high and low ends of the ranking within the same FFN block. We start each deletion from the unpruned checkpoint, independently zero each of the three highest- and three lowest-scored channels per block, and measure the change in correct-answer NLL on the same 36 held-out items. Deleting higher-scored channels raises NLL by 0.0217 more on Gemma and 0.0151 more on Llama. Both 95% block-bootstrap intervals are above zero (Table A). This directly tests score separation within a block. The joint pruning test is in Table D.

W2 / Q2. Probe effect and alternative scores. Relative normalization contributes the larger gain, and the final-state probe adds 3.96/1.30 Mean3 points on Qwen at 40/50% with normalization and selection fixed (Table D). The normalization contrast adds 14.42/5.53. We also apply the same relative normalization and global FFN selector to Wanda and Taylor-CE. Taylor-CE uses cross-entropy saliency with global selection. The original Taylor-FO baseline uses fixed layer quotas. Pooled Qwen sensitivity margins are +2.48/+4.29 points. Table C reports the separate Gemma Q16K full-test comparison.

W3 / Q3. Does the ranking persist with FFN-only pruning? On Qwen, Global-RKU wins all 9/9 Full16K pairs with matched text, 16K-token budget, FFN space, and retained parameters. The margins at 30/40/50% are +0.30/+3.04/+3.12 points. GISP’s means remain higher on Llama and Gemma when attention is fixed and both methods prune FFN channels at common budgets (Table E).

A. Within-block single-channel deletion, mean ΔNLL

ModelTop 3Bottom 3Gap [95% block bootstrap]Positive blocks
Gemma0.0108−0.0108+0.0217 [0.0049, 0.0389]28/42
Llama0.01590.0008+0.0151 [0.0047, 0.0261]23/32

B. Qwen score controls, sensitivity of Mean3 gaps (pp)

The sensitivity calculation adds 2.20 and 4.90 points to the RKU means in bundles 2 and 3. Bundles are equally weighted, with nine pairs each.

SettingRKU − Wanda+RelativeRKU − Taylor-CE+Relative
Bundle 1+6.19+8.15
Bundle 2−0.41+1.26
Bundle 3+1.66+3.45
Pooled+2.48+4.29

C. Gemma Q16K full-test 3-shot score controls, Mean3 gaps (pp)

FFN sparsityRKU − Wanda+RelativeRKU − Taylor-CE+Relative
30%−2.37−0.23
40%−2.10−0.16
50%−2.34−2.18

D. Qwen probe and normalization ablation, Mean3 (%)

Score40%50%
Weighted final-state probe, no relative normalization17.9416.80
Weighted LM-CE probe, relative normalization28.4021.03
Global-RKU32.3622.33

E. Common FFN budgets, three-seed Mean3 (%) at 30/40/50%

BackboneCalibrationGlobal-RKUGISP
Qwen-2.5-7BFull16K50.03/45.49/41.2649.73/42.45/38.14
Llama-3.1-8BFull16K31.17/26.86/21.5338.05/35.11/28.83
Llama-3.1-8BQ16K20.94/18.20/16.3123.56/19.28/16.39
Gemma-2-9BFull16K34.40/29.33/24.5146.36/40.49/35.37
Gemma-2-9BQ16K33.93/27.47/21.9941.32/32.89/27.07
AuthorsOpenReview ↗

A Gentle Reminder of Further Feedback — Reviewer Fe56

Dear Reviewer Fe56,

As the author–reviewer discussion period approaches its conclusion, we would greatly appreciate any further feedback on our detailed rebuttal and revised manuscript.

We have carefully addressed your concerns by adding direct single-channel deletion experiments, comparisons with existing importance scores under the same normalization and global selection procedure, and matched FFN-only evaluations across model families. We have also explicitly acknowledged the settings in which RKU does not outperform GISP.

We hope these additional experiments and clarifications address your principal concerns. We would be grateful to know whether any important questions remain or whether the new evidence changes your assessment.

Thank you again for your time and constructive suggestions.

Best regards,

Authors of Paper 12393

Reviewer_ax72OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of cross-layer importance scoring in global structured pruning for large language models. The authors point out that the Channel Importance Scores of different Transformer layers may exhibit scale discrepancies, making cross-layer comparisons during global pruning unreliable. To address this, the paper proposes Global Relative Kinetic Utility (RKU), which first uses the activation–gradient signals of the final hidden state to estimate the importance of FFN channels, then eliminates the common intra-layer scale via Block-Relative Normalization, and combines this with the Down-Projection Weight Norm for global ranking and pruning. Experiments are conducted on Qwen2.5-7B, Llama3.1-8B, and Gemma2-9B. The results show that RKU can improve the performance of pruned models on some mathematical and non-mathematical reasoning tasks, and demonstrate low-cost recovery and practical deployment results.

Critical strengths

S1. The research problem is of practical significance, and the problem decomposition is relatively clear. This paper distinguishes Channel-Level Credit Assignment from Cross-Layer Score Calibration in global structured pruning, pointing out that effective intra-layer importance ranking does not imply that scores from different layers can be directly compared. This observation is well-motivated and provides a valuable perspective for analyzing the capacity allocation problem in global pruning.

S2. The experiments are relatively comprehensive and include practical deployment validation. The paper evaluates different pruning sparsity levels, mathematical and non-mathematical reasoning tasks, calibration effectiveness across model families, and LoRA Recovery, and reports results for multiple structural random seeds.

Critical weaknesses

W1. The technical novelty of the core method is limited, and the theoretical motivation for cross-layer normalization remains insufficient. The main components of RKU include Activation–Gradient Importance, Weight Norm, Block-Level Normalization, and Global Top-K Selection, all of which are closely related to common designs in existing structured pruning methods. The paper’s main innovation lies in combining these components into a new scoring criterion, but its theoretical analysis only shows that normalization can eliminate a hypothesized Block-Common Multiplicative Scale; it does not prove that such scale discrepancies are the primary source of bias in actual cross-layer importance scoring. In addition, normalization may also eliminate genuine inter-layer importance differences. Can the authors further demonstrate that the Block-Common Scale indeed exists in real models, and explain why the proposed normalization can improve genuine Cross-Layer Importance Comparability rather than merely altering Layer-Wise Sparsity Allocation?

W2. The comparison with the main global pruning baselines has not sufficiently controlled the experimental conditions, which undermines the credibility of the reported performance advantage. In Table 1 of the paper, RKU, GISP, and NIRVANA use different Calibration Corpora, and their pruning structure scopes are also inconsistent. Although the authors supplement the RKU–GISP comparison with the same Calibration Content and Token Budget in Table 14, the two still respectively adopt FFN-Only and MLP+GQA pruning spaces. More importantly, under Full16K Calibration, GISP outperforms RKU at all three sparsity levels; in the partially matched calibration experiments on Llama and Gemma, GISP also performs better. This indicates that RKU’s relative advantage clearly depends on the calibration data and model configuration. Can the authors re-compare the main methods under exactly the same pruning structure space, actual parameter compression ratio, and Calibration Budget, and further explain why RKU falls behind GISP under Full16K and in some cross-model settings?

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ax72

W1. Novelty and cross-layer comparability. Global-RKU calibrates a final-state energy probe for global FFN selection. The reviewer is right that the derivation establishes scale invariance under the stated factorization. Dividing by the block sum cancels a shared positive multiplier when the numerical floor is inactive and preserves the within-block order. The deletion test supports this order at its extremes. Removing high-score channels increases answer NLL by 0.0217 more on Gemma and 0.0151 more on Llama than removing low-score channels (Table C).

Normalization changes retained widths in all 42 Gemma blocks at each tested sparsity (Table B). On Qwen, keeping the probe and selection rule fixed, it adds 14.42/5.53 Mean3 points at 40/50%. That comparison measures the benefit of redistributing width under a shared budget. The NLL test measures within-block ranking. Identifying a true cross-layer scale would require an independent layer-sensitivity measure.

W2. Fair comparison and the Full16K reversal. Table 1 uses published baseline corpora. In the requested matched Full16K comparison, Global-RKU beats GISP on Qwen in all 9/9 pairs. GISP uses C4 in its general-text setting and separately reports a GSM8K calibration setting for math. NIRVANA uses BookCorpus. GISP, Appendix A.3 · NIRVANA

The original Full16K comparison allowed GISP to prune FFN and attention groups. We then fixed the Full16K text, 16K-token budget, coupled FFN space, and exact retained parameter count. The Qwen ranking reverses, with Global-RKU leading by +0.30/+3.04/+3.12 Mean3 points at 30/40/50% (Table A). GISP has higher means on Llama and Gemma at the tested common budgets. Table D shows NIRVANA under the common Gemma FFN budget.

A. Common FFN budgets, three-seed Mean3 (%) at 30/40/50%

BackboneCalibrationGlobal-RKUGISP
Qwen-2.5-7BFull16K50.03/45.49/41.2649.73/42.45/38.14
Llama-3.1-8BFull16K31.17/26.86/21.5338.05/35.11/28.83
Llama-3.1-8BQ16K20.94/18.20/16.3123.56/19.28/16.39
Gemma-2-9BFull16K34.40/29.33/24.5146.36/40.49/35.37
Gemma-2-9BQ16K33.93/27.47/21.9941.32/32.89/27.07

B. Gemma allocation changes after normalization

FFN sparsityBlocks changing widthMask Jaccard
30%42/420.708
40%42/420.659
50%42/420.631

C. Within-block single-channel deletion, mean ΔNLL

ModelTop 3Bottom 3Gap [95% block bootstrap]Positive blocks
Gemma0.0108−0.0108+0.0217 [0.0049, 0.0389]28/42
Llama0.01590.0008+0.0151 [0.0047, 0.0261]23/32

D. Gemma, three-seed Mean3 (%) at 30/40/50%

MethodFull16KQ16K
Global-RKU34.40/29.33/24.5133.93/27.47/21.99
NIRVANA30.97/24.20/19.8132.10/26.45/19.94

Thanks, I decide to maintain my score since it is already positive.

You were the quickest reviewer to respond to our rebuttal, and we were genuinely touched by your prompt and thoughtful engagement. Thank you so much for your time, effort, and support throughout the review process.

Reviewer_yi4MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Global RKU ranks coupled SwiGLU channels across all blocks under one model-wide budget and produces a static pruning topology in a single calibration stage, subject to a minimum retained-width constraint in every block. The ranking uses a final-hidden-state activation–gradient participation signal averaged over calibration tokens, which requires only unlabeled calibration inputs; the scores then undergo block-relative normalisation to mitigate block-common scale and are weighted by the down-projection norm. In the Qwen-2.5-7B signal-factorial ablation, most of the improvement comes from block-relative normalisation, and under questions-only calibration, RKU outperforms GISP at 40% and 50% sparsity.

Critical strengths
  1. The ablation part is well controlled and clearly explains block-relative normalisation as the main source of the gain on Qwen-2.5-7B.

  2. The method is simple: it requires unlabeled calibration inputs and one forward and backward pass per calibration batch, and produces a static pruning topology in a single calibration stage.

  3. The reporting is clear and stable, including different 3 seeds results and physical-deployment measurements on exported dense submodels rather than zero-masked tensors.

Critical weaknesses
  1. It reports a clear advantage over GISP only on Qwen-2.5-7B, while the appendix shows no such advantage on Gemma-2-9B or Llama-3.1-8B. Under the same questions-only calibration, which the paper itself uses for direct RKU–GISP comparison (Sec. 5.3), GISP achieves higher Mean3 than RKU at every sparsity level on both models (Tables 18–19; Mean3 for Gemma computed from Table 21). This suggests that the gains may not generalise beyond a single backbone.

  2. The core idea overlaps with prior work. The activation–gradient score is the first-order Taylor criterion of Molchanov et al. (2017), and the within-block normalisation is close to FLAP (An et al., 2024) and RIA (Zhang et al., 2024). Neither FLAP nor RIA is compared against, and RIA and Molchanov et al. (2017) are not cited.

  3. At high sparsity, the pruned models are close to collapse. At 50% sparsity on Qwen-2.5-7B, RKU reaches only 2.30% on GSM8K, and 26.51% and 30.28% on AQuA and MathQA, where the five-way chance is 20% (Table 1). RKU improves over the other pruning methods, but the practical value of these gains is questionable when performance is already this low. The paper also omits the unpruned models' accuracy, leaving no reference for how much performance has been lost.

Questions for authors
  1. Under questions-only calibration, GISP outperforms RKU on Llama-3.1-8B and Gemma-2-9B. Can the authors explain and show that the advantage over GISP holds for Qwen-2.5-7B?

  2. How does block-relative normalisation differ from the layer-wise standardisation in FLAP and the relative importance in RIA? Could the authors compare against both under the same model-wide budget?

  3. What is the accuracy of the unpruned models on GSM8K, AQuA, and MathQA? Could the authors also report results at lower sparsity (e.g., 10–20%), where the pruned models remain usable?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yi4M

W1. Scope of the advantage. Global-RKU beats GISP in 9/9 Qwen Full16K pairs after matching the text, 16K-token budget, FFN space, and retained parameters (Table A). GISP leads on Llama and Gemma. Among three matched FFN methods on Qwen, RKU also leads in 8/9 ReClor, CommonsenseQA, and LogiQA cells, including all three tasks at 50% (Table E).

W2. Novelty and prior work. The reviewer correctly identifies the shared Taylor form and missing citations. AGF differentiates loss when introducing a dormant feature. Qian et al. apply the absolute activation–loss-gradient product to vision pruning on WideResNet and ResNet with CIFAR-100. RKU differentiates post-RMSNorm final hidden-state energy at unit amplitude for an active SwiGLU channel. The derivative is ∂log⁡γE∣γ=1=⟨Y,∇YE⟩\partial_{\log\gamma}\mathcal E|_{\gamma=1}=\langle Y,\nabla_Y\mathcal E\rangle. It uses the mean absolute activation–gradient product, divides by the floored block sum, and multiplies by the down-projection column norm. With selection and normalization fixed on Qwen, the probe adds 3.96/1.30 Mean3 points over LM-CE at 40/50%. Normalization adds 14.42/5.53 to the weighted energy score. AGF · Vision pruning · Taylor

W3 / Q3. High sparsity, dense references, and low sparsity. The 50% Wiki16K result of 2.30% GSM8K is a real limitation. At the same 50% sparsity on Qwen, Q16K raises mean accuracy across the three math tasks from 19.70% to 39.35%. The new 10% point reaches 76.04% GSM8K with 7.49% fewer total parameters. Table D lists dense and pruned results with their evaluation sets.

Q1. Why Qwen benefits from Q16K. At fixed model, evaluator, 16K budget, and 40% sparsity, seed 42 shows that Qwen’s larger gain comes mainly from GSM8K (+36.09 pp), compared with Gemma (+7.35) and Llama (+0.75). Qwen MathQA also gains 7.64 points (Table C). RKU’s score depends on which channels respond to those calibration questions. The three-seed mean gains from switching RKU to Q16K average 16.96/3.78/1.38 points on Qwen/Gemma/Llama across sparsities. With shared Q16K inputs, RKU leads GISP on Qwen by 3.79/8.61 points at 40/50%. Including solutions at the same token budget increases GISP by 5.38/12.01 points and changes RKU by −0.47/+0.77 (Table F). This pattern is consistent with greater sensitivity to solution text for GISP’s loss-based score. That comparison retains different pruning spaces.

Q2. FLAP and RIA at the same budget. FLAP Raw uses activation variance times the squared down-projection column norm, prior to block z-scoring. RIA Raw aggregates row- and column-relative weights with activation RMS. RIA Relative divides by the block sum. At 7.300/6.652/6.005B retained parameters, RKU beats both Raw controls in 18/18 pairs each, by 9.20/7.30 points on average. RKU leads FLAP Z-score and trails RIA Relative (Table B).

A. Qwen matched Full16K, Mean3 (%)

FFN sparsityRetained paramsRKUGISPWin count
30%5.905B50.0349.733/3
40%5.334B45.4942.453/3
50%4.764B41.2638.143/3

B. Gemma common FFN budgets, Mean3 (%) at 30/40/50%

MethodFull16KQ16K
Global-RKU34.40/29.33/24.5133.93/27.47/21.99
FLAP Raw21.51/19.42/18.2521.16/19.12/16.99
FLAP Z-score29.41/25.64/22.2127.80/22.55/20.64
RIA Raw25.41/21.73/18.2924.81/20.03/17.55
RIA Relative40.18/33.27/28.2838.40/31.71/27.28

C. Qwen corpus diagnostic at 40% (seed 42, 16K tokens, %)

CalibrationGSM8KAQuAMathQA
Wiki16K24.4129.9236.82
Q16K60.5027.9544.46

D. Dense and pruned results (%)

SettingEvaluationGSM8KAQuAMathQAParams cut
Qwen denseFull83.5530.7153.200%
Llama denseFull49.8926.3842.710%
Gemma denseFull68.7635.4348.580%
Qwen 10%Subset76.0430.7152.217.49%
Qwen 20%Subset67.1928.3545.8314.98%
Qwen 50% Wiki16KSubmitted2.3026.5130.2837.44%

E. Qwen, ReClor/CommonsenseQA/LogiQA (%)

FFN sparsityRKUWanda-StructTaylor-FO
30%65.27/73.96/47.4750.60/69.86/43.7856.33/69.40/46.13
40%50.87/62.90/42.1947.00/63.39/38.4043.87/59.30/38.66
50%45.00/50.70/38.1538.00/46.52/33.3335.40/47.50/34.56

F. Qwen with shared calibration text, Mean3 (%)

FFN sparsityQ16K RKU/GISPFull16K RKU/GISP
40%45.96/42.1745.49/47.55
50%39.35/30.7440.12/42.75
Reviewer_yi4MOpenReview ↗

Review

Thank you for your answers. The authors addressed some of my concerns, however, some remain.

  1. The generalisation problem is not well addressed. The proposed 'fix' is a methodologically suspect: they restrict GISP to FFN-only pruning space to match RKU, then declare a 9/9 win on Qwen. But this artificially handicaps GISP, which was designed to prune both FFN and attention components. Winning a constrained version of your competitor is not evidence of superiority.
  2. Regarding the novelty, the authors distinguish RKU from FLAP and RIA through implementation details, not principled motivation
AuthorsOpenReview ↗

Response to Reviewer yi4M

Thank you for following up.

1. GISP and generalization. The original submission already reports runs in which GISP prunes both FFN channels and attention groups (§5.3 and Appendix C, original Table 14, now Table 15). With the same questions-only calibration (Q16K), RKU leads on Qwen by 3.79 and 8.61 Mean3 points at the 40% and 50% operating points. With Full16K, GISP leads at all three.

We added the FFN-only experiment (revision §5.3, Table 6) to compare the subnetworks each method selects when attention is held fixed and both models retain exactly the same number of parameters. GISP still uses its loss-based score and recomputes it after each pruning step. Under Full16K, RKU leads in all nine Qwen seed–sparsity pairs. At the common 30–50% FFN budgets on Llama and Gemma, GISP has higher mean scores.

2. Novelty. Alternating Gradient Flows (Kunin et al., NeurIPS 2025) is a theory of how dormant features become active during training. Let fAf_A be the current predictor, fif_i a dormant feature, and λ\lambda its amplitude. For squared loss, AGF defines

=\mathbb E_x\langle f_i(x),y(x)-f_A(x)\rangle.$$ $U_i$ is the initial rate of loss reduction from adding feature $i$. For small positive $\lambda$, the loss drops by approximately $\lambda U_i$ when $U_i>0$: the feature aligns with the current prediction residual. **Near a saddle, this utility controls the growth of the dormant neuron’s parameter norm.** When the leading order is quadratic ($\kappa=2$), $d\log\\|\theta_i\\|/dt\approx 2\eta U_i$. AGF accumulates the utility over training to predict when the neuron crosses the activation threshold. Qian et al. (2026) used $\mathbb E|Y_c\odot\nabla_{Y_c}\mathcal L_{\rm cls}|$ to score channels in image classifiers. **RKU asks which active channels a frozen LLM should retain.** We scale a SwiGLU channel by $\gamma$ before the down projection: $\gamma=0$ silences the channel, and the derivative at $\gamma=1$ measures the local response to rescaling it from its current amplitude. We probe the energy of the post-RMSNorm representation that feeds the output head, so the score can capture the response through the rest of the model using calibration text alone: $$\mathcal E=\lVert H_{\mathrm{post}}\rVert_F^2,\qquad \left.\frac{\partial\mathcal E}{\partial\log\gamma}\right|_{\gamma=1} =\sum_t Y_{l,c,t}\\,\partial_{Y_{l,c,t}}\mathcal E.$$ Opposite responses at different token positions can cancel in this sum. We take their magnitudes before averaging. Dividing by total block participation cancels a shared multiplicative scale when the numerical floor is inactive, so it cannot drive the global allocation of FFN width. The outgoing weight norm enters separately: $$A_{l,c}=\mathbb E_x\operatorname{mean}_t \left|Y_{l,c,t}\\,\partial_{Y_{l,c,t}}\mathcal E\right|,\qquad S_{l,c}=\lVert W_{\mathrm{down}}^{(l)}[:,c]\rVert_2 \frac{A_{l,c}}{\max(\sum_j A_{l,j},10^{-12})}.$$ For the same down projection $W$ and input channel $c$, FLAP's channel score and RIA's weight score are $$F_c=\operatorname{Var}(Y_c)\lVert W_{:c}\rVert_2^2, \qquad R_{ic}=|W_{ic}| \left(\frac{1}{\lVert W_{i:}\rVert_1} +\frac{1}{\lVert W_{:c}\rVert_1}\right) \lVert Y_c\rVert_2^a.$$ Our FFN RIA control uses activation RMS and sums the weight scores along each column. If $W=I$ and two channels have identical activations, FLAP and RIA tie them; normalization preserves the tie. **RKU can rank them differently because the downstream energy gradient can differ even when these local quantities agree.** Original §5.4, Table 6 (now Table 7) isolates the probe: switching from next-token cross-entropy to terminal energy gains **3.96 and 1.30 Mean3 points** on Qwen at 40% and 50% FFN sparsity, with weighting, normalization, and selection fixed. **Qwen-2.5-7B: RKU minus GISP in three-seed Mean3 (percentage points)** | Calibration and GISP pruning space | 30% | 40% | 50% | | --- | ---: | ---: | ---: | | Full16K, FFN only, exact parameter match | **+0.30** | **+3.04** | **+3.12** | | Q16K, FFN + attention | −0.98 | **+3.79** | **+8.61** | | Full16K, FFN + attention | −0.37 | −2.06 | −2.63 | The joint-pruning rows use the original nominal operating points; only the first row matches retained parameters exactly. Mean3 averages GSM8K, AQuA-RAT, and MathQA accuracy. **Qwen-2.5-7B: controlled score ablation (Mean3, %)** | Score | 40% FFN sparsity | 50% FFN sparsity | | --- | ---: | ---: | | Weighted final-state energy probe, without block normalization | 17.94 | 16.80 | | Weighted next-token cross-entropy probe, with block normalization | 28.40 | 21.03 | | Global-RKU: weighted final-state energy probe, with block normalization | **32.36** | **22.33** |
AuthorsOpenReview ↗

A Gentle Follow-up on Our Clarifications — Reviewer yi4M

Dear Reviewer yi4M,

Thank you again for your follow-up comments and for taking the time to engage with our rebuttal.

As the discussion period approaches its conclusion, we would greatly appreciate any further feedback on our latest response, particularly regarding the purpose of the matched FFN-only comparison and the methodological distinction between RKU and existing importance criteria.

We have clarified that the FFN-only experiment is intended to isolate channel selection under a common pruning budget, rather than establish universal superiority over GISP. We have also provided a more detailed mathematical explanation of the final-state energy probe and its controlled ablation.

We would be grateful to know whether these clarifications have addressed your remaining concerns, or whether any important issues remain unresolved.

Thank you again for your time and consideration.

Best regards,

Authors of Paper 12393

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thanks for All Reviewers

We thank all the reviewers and the area chair for their careful reading and constructive suggestions. Your feedback prompted us to compare methods at matched FFN budgets, test alternative scores under the same selector, and add dense and low-sparsity results. We marked the manuscript revisions in blue. The separate Revision Change Summary lists the changes and their supporting results.

AuthorsOpenReview ↗

Revision Change Summary

The revised manuscript marks changes in blue. The main text ends on page 10.

[G1] Matched FFN comparison added to §5.3, Table 6. With the same Full16K text, token budget, pruning space, and retained parameters, Global-RKU leads GISP on Qwen in all 9/9 pairs. GISP has higher means on Llama and Gemma under the common FFN budgets.

[G2] Stronger baselines and dense references. We report FLAP, RIA, and NIRVANA at common Gemma FFN budgets. RKU leads FLAP Raw and RIA Raw in 18/18 comparisons each. We add dense full-test references for all three models and report 10%/20% Qwen results on fixed subsets.

[G3] Scoring and normalization. We add same-selector Wanda and Taylor-CE controls. On Qwen, relative normalization adds 14.42/5.53 Mean3 points at 40%/50% sparsity, and the final-state probe adds 3.96/1.30 with normalization and selection fixed. A single-channel deletion test finds greater mean NLL increases for higher-scored channels within a block on Gemma and Llama.

[G4] Method and conclusion. We trace the score from AGF through prior vision pruning, explain its final-state energy objective and block calibration, and clarify the conclusions supported on each backbone. We add the missing related-work citations.

Matched Qwen-2.5-7B, Full16K (three-seed Mean3 %)

FFN sparsityRetained parametersGlobal-RKUGISPMargin (pp)Wins
30%5.905B50.0349.73+0.303/3
40%5.334B45.4942.45+3.043/3
50%4.764B41.2638.14+3.123/3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.