acceptodds
Under review as a conference paper at ICLR 2027

NS-UDPD: A Finite-Domain Joint Selection Method for Layer-Wise Sparsity Allocation in LLM Pruning

Abstract

Layer-wise sparsity allocation improves unstructured post-training pruning of large language models (LLMs) by distributing a global sparsity budget across Transformer layers. Existing metric-based methods overlook pruning-cost variations across sparsity levels. Search-based and learning-based methods lack global optimality guarantees for discrete allocations, while reconstruction-based methods optimize Euclidean error rather than the language-modeling negative log-likelihood (NLL) objective. We therefore propose **NS-UDPD, a finite-domain joint selection method** that selects one candidate sparsity level per layer while exactly matching the target mean layer sparsity. **Diagonal-Kronecker NLL Sensitivity (DKNS)** combines squared weights, input-activation second moments, and squared output-NLL-gradient energies into per-weight DKNS proxy deletion costs, which candidate deletion masks aggregate into layer-level DKNS state costs. Cost-Only Allocation may concentrate state costs in shallow layers, where pruning perturbations propagate through subsequent layers. **Uniform Depth-Prefix Dominance (UDPD)** therefore bounds the cumulative state cost at every depth prefix using the uniform reference allocation. A three-level lexicographic objective first minimizes the full-depth DKNS state cost and then improves the depth-prefix cost distribution and layer-wise sparsity balance. We evaluate NS-UDPD on five models from the LLaMA-3 and Qwen3.5 series under three fixed local pruning rules and 40%–70% global unstructured sparsity. At 70% sparsity, it reduces WikiText-2 perplexity relative to the best competing allocation on Qwen3.5-2B with SparseGPT by 31.55% and raises six-task mean accuracy from 37.20% to 41.26% on Qwen3.5-4B with Wanda relative to Uniform.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM Pruning, Layer-wise Sparsity Allocation, Unstructured Sparsity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_j7pFOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates layer-wise sparsity allocation as discrete optimization over candidate pruning states. DKNS assigns gradient-weighted deletion costs, UDPD constrains cumulative costs relative to uniform allocation, and a lexicographic MILP selects the allocation. Experiments cover five models, three local pruning rules, four sparsity levels, and six downstream tasks. The claimed contribution is a combination of NLL-sensitive cost modeling, depth-prefix constraints, and exact optimization of the resulting surrogate.

Critical strengths
  • Substantial performance improvements in several settings. Tables 1 and 5 show the best PPL in most configurations.
  • Practical deployment evidence. The SpInfer experiments demonstrate that the resulting non-uniform sparse models can achieve end-to-end acceleration, reaching approximately 1.68–1.69× speedup at 70% sparsity on the two evaluated OPT models.
  • Feasible and flexible prefix constraints. UDPD preserves the uniform allocation as a feasible reference while allowing non-monotonic sparsity profiles. This provides a structured way to protect shallow layers without prescribing a fixed depth-dependent schedule.
Critical weaknesses
  • The experiments do not establish the contribution of the proposed components. Section 3.3 varies only grid resolution and candidate range; there is no comparison against DKNS Cost-Only Allocation, an otherwise identical allocator with G^=1\widehat G=1, or removal of the secondary objectives. Consequently, the results do not establish whether gradient weighting, UDPD, or exact selection explains the gains. A controlled component ablation and a comparison between predicted costs and measured held-out NLL changes are necessary, particularly for SparseGPT, whose compensation updates are excluded from the state cost.
  • Downstream superiority over competitive allocators is much weaker than the headline suggests. In Table 2, Qwen3.5-4B/Wanda improves by 4.06 percentage points over Uniform, but only 0.14 points over DLP; Qwen3.5-2B/SparseGPT improves by 0.09 points over ATP. Conversely, Qwen3.5-4B/SparseGPT obtains 43.54% versus ATP’s 45.23%, despite achieving better PPL. With one calibration seed and no uncertainty estimates, these results support substantial improvements over Uniform in selected settings, but do not establish reliable downstream superiority over strong allocators. Repeated calibration samples and paired uncertainty estimates would clarify this distinction.
  • The methodological novelty requires clearer clarification. Constrained layer-wise allocation using precomputed costs already appears in SPDY[1], while protecting early layers against propagated pruning errors motivates ATP[2]. The paper should clarify this distinction and demonstrate the additional benefit of UDPD over cost-only allocation and established depth-aware allocation strategies.

References

[1] Frantar et al. SPDY: Accurate Pruning with Speedup Guarantees

[2] Huang et al. Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

Questions for authors
  1. What are the actual parameter-weighted sparsities and retained-weight counts for each Qwen3.5 allocation, and do the advantages persist when these budgets are matched?
  2. How much does UDPD improve held-out PPL over DKNS Cost-Only Allocation, and does DKNS predict actual pruning-induced NLL changes better than the corresponding activation-only cost?
8 comments
AuthorsOpenReview ↗

Response to W1 and Q2 (Reviewer j7pF)

Thank you for your constructive feedback and for highlighting NS-UDPD’s ppl improvements and the flexibility of its depth-prefix constraints. We respond to your comments point by point below.

Response to W1 and Q2 (Reviewer j7pF)

Thank you for asking about each design's contribution. We conduct three controlled ablations on Qwen3.5-4B at a target mean layer sparsity of 70% and analyze how DKNS costs rank actual NLL changes. We will add these experiments results to Appendix G of the revised manuscript. In the tables, ✓/✗ means UDPD prefix constraints are enabled/disabled or the stage is run/skipped.

1. The Three DKNS Factors

With Wanda, we test all seven nonempty combinations of WW, AA, and GG, along with Uniform as a control. WW is the squared weight wℓ,m,i,j2w_{\ell,m,i,j}^{2}, AA the input-activation second moment A^ℓ,m,j\widehat{A}_{\ell,m,j}, and GG the squared output-NLL-gradient energy G^ℓ,m,i\widehat{G}_{\ell,m,i}. All seven combinations use the same calibration samples and follow the same Wanda rule for generating candidate deletion masks. For each, we set omitted factors to 1, compute uniform prefix bounds from its costs, and run all three stages. Uniform targets 70% sparsity in every layer. Table 1 reports the results.

Under the same allocation procedure, restoring GG to WAWA, AA to WGWG, and WW to AGAG lowers PPL from 107.34, 234.79, and 48.79 to 38.50 and raises Mean by 4.55, 1.85, and 1.17 percentage points, respectively. Both metrics improve in each comparison. The WAWA–WAGWAG comparison supports collecting output-gradient statistics during allocation.

DKNS Costs and Actual NLL Changes. For each pruner, we score the same model-wide allocations using full-depth costs from DKNS (WAGWAG) and its activation-weighted variant (WAWA, G=1G=1) and measure their full-model NLL changes. Spearman’s ρ\rho is the Pearson correlation between cost ranks and NLL-change ranks. Switching from WAWA to WAGWAG raises ρ\rho from 0.214 to 0.976 for Wanda and from −0.286 to 0.952 for SparseGPT. In both settings, full DKNS cost ranks agree more closely with actual NLL-change ranks than those of WAWA.

Table 1. DKNS factor ablation on Qwen3.5-4B with Wanda.

VariantUDPDStage 1Stage 2Stage 3PPL ↓Mean (%) ↑
Uniform✗✗✗✗98.1837.20
W✓✓✓✓94.5937.26
A✓✓✓✓152.5736.18
G✓✓✓✓369.6338.02
WA✓✓✓✓107.3436.71
WG✓✓✓✓234.7939.41
AG✓✓✓✓48.7940.09
WAG✓✓✓✓38.5041.26

2. Cost-Only Comparison with Fixed DKNS Costs

For each pruner (Wanda or SparseGPT), both variants share the candidate set and full DKNS (WAGWAG) cost table. NS-CostOnly disables UDPD prefix constraints and runs only Stage 1 to minimize the full-depth DKNS state cost. Full NS-UDPD enforces these constraints in all three stages.

Table 2 reports the results. Compared with NS-CostOnly, full NS-UDPD reduces PPL from 53.72 to 38.50 with Wanda and from 29.28 to 25.85 with SparseGPT, raising Mean by 0.27 and 0.72 percentage points, respectively. These results show the combined benefit of UDPD prefix constraints and the secondary optimization objectives.

Table 2. NS-UDPD versus NS-CostOnly with fixed WAG costs.

PrunerVariantDKNS CostUDPDStage 1Stage 2Stage 3PPL ↓Mean (%) ↑
WandaNS-CostOnlyWAG✗✓✗✗53.7240.99
WandaNS-UDPDWAG✓✓✓✓38.5041.26
SparseGPTNS-CostOnlyWAG✗✓✗✗29.2842.82
SparseGPTNS-UDPDWAG✓✓✓✓25.8543.54

3. Secondary Optimization Objectives

For Wanda and SparseGPT, we fix full DKNS (WAGWAG) costs and run the stages sequentially under UDPD prefix constraints. Stage 1 minimizes the full-depth DKNS state cost. Stage 2 preserves that optimum within numerical tolerance and minimizes the worst normalized depth-prefix ratio relative to the uniform reference allocation. Stage 3 preserves both prior optima within their respective tolerances and minimizes layer-wise sparsity variance.Table 3 reports the results.

For Wanda and SparseGPT, respectively, Stage 2 lowers PPL by 7.33 and 5.17 and raises Mean by 0.32 and 0.43 percentage points. Stage 3 further lowers PPL by 2.13 and 0.45 and raises Mean by 0.11 and 0.36 percentage points. Together, Stages 2 and 3 add 2.175 s and 0.232 s of MILP solve time over Stage 1 alone, respectively. The measured gains and small additional solve times support retaining both objectives.

Table 3. Three-stage ablation with fixed WAG costs and UDPD prefix constraints.

PrunerDKNS CostUDPDStage 1Stage 2Stage 3PPL ↓Mean (%) ↑Cumulative MILP Time (s)
WandaWAG✓✓✗✗47.9640.831.057
WandaWAG✓✓✓✗40.6341.152.165
WandaWAG✓✓✓✓38.5041.263.232
SparseGPTWAG✓✓✗✗31.4742.750.186
SparseGPTWAG✓✓✓✗26.3043.180.306
SparseGPTWAG✓✓✓✓25.8543.540.418
AuthorsOpenReview ↗

Response to W2 (Reviewer j7pF)

Response to W2 (Reviewer j7pF)

Thank you for suggesting paired reruns to assess the effect of calibration samples.

At a target mean layer sparsity of 70%, we repeat each comparison using five C4 calibration seeds. For each seed, we sample 128 sequences at each calibration stage. We report the mean accuracy across six zero-shot tasks (Mean) and WikiText-2 perplexity (PPL). For each method, the mean ± sample standard deviation (SD) summarizes its average performance and variability across the five seeds. Δ\Delta denotes the mean paired difference, computed as NS-UDPD minus the baseline. Brackets show two-sided 95% paired tt confidence intervals (CIs), which quantify uncertainty in the estimated mean difference due to calibration sampling. Here, pp denotes percentage points.

Table 4. Paired rerun results for five C4 calibration seeds at a target mean layer sparsity of 70%.

Model + PrunerBaselineBaseline Mean (%) ↑NS-UDPD Mean (%) ↑Baseline PPL ↓NS-UDPD PPL ↓Δ\Delta Mean (pp) [95% CI]Δ\Delta PPL [95% CI]
Qwen3.5-4B + WandaDLP41.16 ± 0.1441.35 ± 0.0991.89 ± 0.1338.54 ± 0.07+0.19 [0.12, 0.27]−53.35 [−53.43, −53.27]
Qwen3.5-2B + SparseGPTATP39.15 ± 0.1239.30 ± 0.1164.92 ± 0.1544.37 ± 0.10+0.15 [0.07, 0.23]−20.55 [−20.74, −20.36]
Qwen3.5-4B + SparseGPTATP44.98 ± 0.1843.79 ± 0.1635.66 ± 0.0925.88 ± 0.05−1.19 [−1.54, −0.83]−9.78 [−9.89, −9.66]

Across the six zero-shot tasks, NS-UDPD improves mean accuracy by 0.19 percentage points over DLP with Qwen3.5-4B + Wanda and by 0.15 percentage points over ATP with Qwen3.5-2B + SparseGPT. Both paired confidence intervals lie entirely above zero, supporting a mean accuracy advantage in these two settings across different calibration samples.

In the WikiText-2 language modeling evaluation, the 95% confidence intervals for the paired PPL differences lie entirely below zero in all three comparisons, supporting lower mean PPL for NS-UDPD in all three settings. NS-UDPD also has a lower sample SD for PPL than the corresponding baseline in each setting, indicating less PPL variability across these five sets of calibration samples.

These results suggest that NS-UDPD outperforms the baselines in most experimental configurations and tasks. We will report these results in Appendix J of the revised manuscript.

AuthorsOpenReview ↗

Response to W3 (Reviewer j7pF)

Response to W3 (Reviewer j7pF)

Thank you for pointing out the connections between NS-UDPD, SPDY, and ATP.

SPDY [1] uses local search to estimate layer-wise error scores and dynamic programming to select layer-wise sparsities that minimize accuracy loss subject to a target inference speedup. ATP [2] sets layer-wise sparsities as an arithmetic progression that increases with depth and searches for the common difference. The lower sparsities assigned to shallow layers mitigate the propagation and amplification of reconstruction errors in subsequent layers. NS-UDPD exactly matches the target mean layer sparsity and separately bounds the cumulative DKNS state cost at every depth prefix. At each depth dd, allocation xx must satisfy Ad(x)≤UdA_d(x)\leq U_d, where Ad(x)A_d(x) and UdU_d are the cumulative DKNS state costs over the first dd layers under xx and Uniform at the same budget, respectively. Bounding each prefix prevents cost savings in deeper layers from offsetting excess costs in shallower prefixes. If the cumulative state cost of the first d−1d-1 layers is below its bound, the DKNS state cost of layer dd may exceed that layer's cost under Uniform by up to Ud−1−Ad−1(x)U_{d-1}-A_{d-1}(x). This slack gives the joint optimization room to adjust layer-wise sparsities while exactly meeting the budget and satisfying all prefix bounds.

We use SparseGPT on Qwen3.5-4B at a fixed target mean layer sparsity of 70%. All variants share the DKNS cost table and candidate grid. As shown in Table 5, Cost-Only minimizes only the full-depth DKNS state cost AL(x)A_L(x). UDPD (Uniform) constructs prefix bounds using 70% sparsity in every layer. ATP-shaped projects the ATP sparsity schedule, which increases with depth, onto the same grid while preserving the budget, then constructs prefix bounds from the projected sparsities. DKNS-ranked reorders the same projected sparsities, assigning higher sparsities to layers with smaller DKNS cost increases when sparsity is raised by one grid step from SS. Each of the three reference-based variants jointly selects layer-wise sparsities under its own prefix bounds, sequentially minimizing the full-depth DKNS state cost, the worst normalized depth-prefix ratio, and the layer-wise sparsity variance.

Compared with Cost-Only, which minimizes only the full-depth DKNS state cost, full UDPD (Uniform) reduces WikiText-2 PPL from 29.28 to 25.85 and increases six-task mean accuracy from 42.82% to 43.54%. At the same budget and with the same solver stages, UDPD (Uniform) reduces PPL by 22.12% relative to ATP-shaped and by 18.27% relative to DKNS-ranked, while increasing six-task mean accuracy by 0.88 and 0.81 percentage points, respectively. ATP-shaped assigns lower reference sparsities to shallow layers, which may tighten the corresponding prefix bounds and limit how much pruning less sensitive shallow layers can accommodate. DKNS-ranked orders the entire set of layer sparsities using one-step DKNS cost increases at SS, and this ordering may vary across sparsity levels. These results support using Uniform as the default reference in this setting and show the benefit of full UDPD over minimizing the full-depth DKNS state cost alone. We will include this analysis in Appendix H of the revised manuscript.

Table 5. Comparison of allocation strategies on Qwen3.5-4B with SparseGPT at a target mean layer sparsity of 70%. ✓ and ✗ indicate enabled and disabled, respectively.

Allocation StrategyPrefix ConstraintsStage 1Stage 2Stage 3WikiText-2 PPL ↓Six-Task Mean (%) ↑
Cost-Only✗✓✗✗29.2842.82
UDPD (Uniform)✓✓✓✓25.8543.54
ATP-shaped✓✓✓✓33.1942.66
DKNS-ranked✓✓✓✓31.6342.73

[1]Frantar, Elias, and Dan Alistarh. "SPDY: Accurate pruning with speedup guarantees." International conference on machine learning. PMLR, 2022.

[2]Huang, Weizhong, et al. "Determining layer-wise sparsity for large language models through a theoretical perspective." arXiv preprint arXiv:2502.14770 (2025).

Reviewer_ggeWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author proposes NS-UDPD, a method for choosing layer-wise sparsity levels in post-training LLM pruning. It constructs a deletion-cost proxy, DKNS, from weight magnitudes, activation statistics, and output gradients of the language-modeling loss. A discrete optimization then selects sparsity levels while matching the target mean layer sparsity and keeping cumulative costs at each depth prefix below those of uniform allocation. Experiments evaluate five models and three local pruning rules at 40%–70% sparsity, reporting substantial perplexity improvements alongside additional zero-shot and inference-speed evaluations.

Critical strengths

S1. The perplexity results are strong across multiple settings: NS-UDPD wins 24 of the 30 comparisons at 60%–70% sparsity, including all ten SparseGPT settings, suggesting that the allocation strategy is useful beyond a single model or pruning rule.

S2. The depth-prefix formulation offers a clear way to control where pruning costs accumulate without requiring a monotonic sparsity schedule, while retaining a feasible uniform reference and explicitly stating the conditions for exact lexicographic optimization.

Critical weaknesses

W1. The main design choices are not separately validated. Section 3.3 varies only the sparsity grid and candidate range; it does not compare DKNS against a cost without output-gradient weighting, UDPD against cost-only allocation, or the full objective against its first stage alone. Appendix A.2 appropriately acknowledges that DKNS is a separable sensitivity proxy rather than the actual NLL change, so solving this proxy exactly does not itself establish the benefit of the proposed weighting and prefix constraints. Controlled component ablations are needed to explain what drives the gains and justify the method’s added complexity.

W2. Matching mean layer sparsity does not necessarily match the actual compression budget. Equation 2 enforces an unweighted average of layer sparsities, whereas global sparsity over prunable weights is ∑ℓnℓsℓ/∑ℓnℓ\sum_\ell n_\ell s_\ell/\sum_\ell n_\ell, where nℓn_\ell is the number of prunable weights in layer ℓ\ell. This distinction needs checking for Qwen3.5: its official configuration mixes linear-attention and full-attention blocks, so equal prunable layer sizes should not simply be assumed. The paper should specify the pruned modules and report achieved parameter-weighted sparsity for each method; otherwise, the comparisons labeled as matching “global sparsity” could retain different numbers of weights.

Questions for authors

Q1. Do you have existing results separating the contribution of DKNS’s output-gradient weighting from that of UDPD, particularly a comparison against cost-only allocation using the same DKNS costs and candidate set?

Q2. Are the prunable parameter counts identical across the blocks used in your Qwen3.5 experiments? If not, what actual parameter-weighted sparsities do NS-UDPD and the baselines achieve at the reported targets?

Q3. Which MILP solver, optimality gaps, and stage-locking tolerances ϵ1,ϵ2\epsilon_1,\epsilon_2 were used, and do the second and third stages actually change the selected allocation? Please also provide the measured allocation overhead, including gradient collection, candidate-cost construction, and solver time.

I could move to weak accept if the rebuttal confirms comparable actual compression budgets and provides convincing evidence that the main components contribute beyond cost-only allocation.

5 comments
AuthorsOpenReview ↗

Response to W1 and Q1(Reviewer ggeW)

Thank you for your constructive feedback and for highlighting NS-UDPD’s strong perplexity results across settings and the design of its depth-prefix constraints. We respond to your comments point by point below.

Response to W1 and Q1(Reviewer ggeW)

Thank you for highlighting the need to isolate each design's contribution. We run three controlled ablations on Qwen3.5-4B at a target mean layer sparsity of 70% to assess the DKNS factors, full NS-UDPD versus NS-CostOnly, and the secondary objectives. We will add these ablations to Appendix G of the revised manuscript and specify what each comparison tests. In the tables, ✓/✗ means the UDPD prefix constraints are enabled/disabled or the stage is run/skipped. PPL is WikiText-2 perplexity and Mean is six-task zero-shot mean accuracy.

1. The Three DKNS Factors

With Wanda, we test all seven nonempty combinations of WW, AA, and GG and the Uniform control. WW is the squared weight wℓ,m,i,j2w_{\ell,m,i,j}^{2}, AA the input-activation second moment A^ℓ,m,j\widehat{A}_{\ell,m,j}, and GG the squared output-NLL-gradient energy G^ℓ,m,i\widehat{G}_{\ell,m,i}. All seven combinations use the same calibration samples and follow the same Wanda rule for generating candidate deletion masks. For each, we set omitted factors to 1, compute uniform prefix bounds from its costs, and run all three stages. Uniform assigns 70% sparsity to every layer. Table 1 reports the results.

Under the same allocation procedure, adding GG to WAWA, AA to WGWG, and WW to AGAG reduces PPL from 107.34, 234.79, and 48.79, respectively, to 38.50 and increases six-task zero-shot mean accuracy by 4.55, 1.85, and 1.17 percentage points, respectively. In this setting, restoring any one of the omitted factors improves both metrics. In particular, the comparison between WAWA and WAGWAG supports collecting output-gradient statistics during allocation.

Table 1. DKNS factor ablation on Qwen3.5-4B with Wanda.

VariantUDPDStage 1Stage 2Stage 3PPL ↓Mean (%) ↑
Uniform✗✗✗✗98.1837.20
W✓✓✓✓94.5937.26
A✓✓✓✓152.5736.18
G✓✓✓✓369.6338.02
WA✓✓✓✓107.3436.71
WG✓✓✓✓234.7939.41
AG✓✓✓✓48.7940.09
WAG✓✓✓✓38.5041.26

2. Cost-Only Comparison with Fixed DKNS Costs

For both Wanda and SparseGPT, NS-CostOnly and full NS-UDPD share the candidate set and full DKNS (WAGWAG) cost table. NS-CostOnly disables UDPD prefix constraints and runs only Stage 1 to minimize the full-depth DKNS state cost. Full NS-UDPD enforces these constraints in all three stages. Table 2 reports the results.

Compared with NS-CostOnly, full NS-UDPD reduces PPL from 53.72 to 38.50 with Wanda and from 29.28 to 25.85 with SparseGPT. It also improves six-task zero-shot mean accuracy by 0.27 and 0.72 percentage points, respectively. These results show the combined benefit of the UDPD prefix constraints and the secondary optimization objectives.

Table 2. Comparison of NS-UDPD and NS-CostOnly with fixed WAG costs.

PrunerVariantDKNS CostUDPDStage 1Stage 2Stage 3PPL ↓Mean (%) ↑
WandaNS-CostOnlyWAG✗✓✗✗53.7240.99
WandaNS-UDPDWAG✓✓✓✓38.5041.26
SparseGPTNS-CostOnlyWAG✗✓✗✗29.2842.82
SparseGPTNS-UDPDWAG✓✓✓✓25.8543.54

3. Secondary Optimization Objectives

With Wanda and SparseGPT, we fix full DKNS (WAGWAG) costs and run all three stages sequentially under the UDPD prefix constraints. Stage 1 minimizes the full-depth DKNS state cost. Stage 2 preserves the Stage 1 optimum within numerical tolerance and minimizes the worst normalized depth-prefix ratio relative to the uniform reference allocation. Stage 3 preserves both preceding optima within their respective tolerances and minimizes layer-wise sparsity variance. Table 3 reports the results.

Adding Stage 2 reduces PPL by 7.33 with Wanda and 5.17 with SparseGPT and increases six-task zero-shot mean accuracy by 0.32 and 0.43 percentage points, respectively. Adding Stage 3 further reduces PPL by 2.13 and 0.45 and increases six-task zero-shot mean accuracy by another 0.11 and 0.36 percentage points, respectively. Relative to Stage 1 alone, the two secondary stages together add 2.175 s of MILP solve time with Wanda and 0.232 s with SparseGPT. The measured performance gains and the small additional solve times support retaining both secondary objectives.

Table 3. Three-stage ablation with fixed WAG costs and UDPD prefix constraints.

PrunerDKNS CostUDPDStage 1Stage 2Stage 3PPL ↓Mean (%) ↑Cumulative MILP Time (s)
WandaWAG✓✓✗✗47.9640.831.057
WandaWAG✓✓✓✗40.6341.152.165
WandaWAG✓✓✓✓38.5041.263.232
SparseGPTWAG✓✓✗✗31.4742.750.186
SparseGPTWAG✓✓✓✗26.3043.180.306
SparseGPTWAG✓✓✓✓25.8543.540.418
AuthorsOpenReview ↗

Response to W2 and Q2(Reviewer ggeW)

Response to W2 and Q2(Reviewer ggeW)

Thank you for pointing out the distinction between the target mean layer sparsity and the actual pruning budget. Using Wanda on Qwen3.5-4B, we measured the actual parameter-weighted sparsity and WikiText-2 PPL of six allocation methods at the 60% and 70% targets.

Pruning scope. We prune the projection weights of proj_qkv, proj_z, proj_a, proj_b, and proj_out in linear attention blocks, proj_q, proj_k, proj_v, and proj_o in full attention blocks, and proj_gate, proj_up, and proj_down in the MLPs of both block types. Each of the 24 linear attention blocks contains 112,885,760 prunable weights, and each of the 8 full attention blocks contains 107,479,040, for a total of N=3,569,090,560N=3{,}569{,}090{,}560. Eq. (2) in the manuscript constrains the arithmetic mean of the assigned layer sparsities to SS. Let nℓn_\ell be the number of prunable weights in layer ℓ\ell and ZℓZ_\ell the number of those weights that are zero after pruning. The actual parameter-weighted sparsity is ∑ℓZℓ/∑ℓnℓ=Z/N\sum_\ell Z_\ell/\sum_\ell n_\ell=Z/N, and the number of retained nonzero weights is N−ZN-Z.

Table 4. Actual parameter-weighted sparsity and WikiText-2 PPL for Qwen3.5-4B with Wanda.

Method60% Actual Sparsity60% PPL70% Actual Sparsity70% PPL
Uniform59.99725%21.4569.99908%98.18
OWL59.97942%20.5369.97925%78.19
DLP59.98683%23.1369.98862%91.86
LSA59.98803%21.1969.99162%73.65
ATP59.96688%29.5269.96520%97.26
NS-UDPD60.04020%17.3370.04760%38.50

Measured results. As shown in Table 4, for every method, the actual parameter-weighted sparsity differs from the target mean layer sparsity by less than 0.05 percentage points in absolute terms. These deviations do not affect the pruning algorithm's performance. Uniform assigns the same sparsity to every layer, and its deviation arises because Wanda rounds down the pruning quota for each row. OWL, ATP, DLP, and LSA assign different sparsities to the two types of attention blocks. Because each linear attention block contains more prunable weights, its sparsity receives greater weight in the parameter-weighted average. Both parameter weighting and Wanda's per-row rounding down contribute to the deviations for these four baselines, whose actual sparsities are all slightly below the targets. NS-UDPD assigns higher sparsity to linear attention blocks than to full attention blocks. Under the combined effects of parameter weighting and Wanda's per-row rounding down, its actual sparsity exceeds the 60% and 70% targets by 0.0402 and 0.0476 percentage points, respectively. At both targets, NS-UDPD achieves higher actual parameter-weighted sparsity and lower WikiText-2 PPL than every baseline. These results support the performance advantage of NS-UDPD on Qwen3.5, which uses both linear and full attention. We will add the results for each method reported in this response to Appendix F of the revised manuscript.

AuthorsOpenReview ↗

Response to Q3 (Reviewer ggeW)

Response to Q3 (Reviewer ggeW)

Thank you for asking about the solver settings and allocation overhead. We use SparseGPT on LLaMA-3.1-8B at a target mean layer sparsity of 50%. We solve all three mixed-integer linear programming (MILP) stages with HiGHS via SciPy's milp interface, using a relative optimality gap of 10−610^{-6} at each stage. The first stage lock applies to the full-depth DKNS state cost, with a tolerance of 1.24405×10−91.24405\times10^{-9} on the normalized scale. The second stage lock applies to the worst normalized depth-prefix ratio, with a tolerance of ϵ2=2.00×10−9\epsilon_2=2.00\times10^{-9}. Both Stage 2 and Stage 3 changed the selected allocation. Stage 2 changed the sparsity of 3 of the 32 layers, and Stage 3 changed the sparsity of 2 more layers. Table 5 reports the runtime for each step. The three MILP stages took 2.155 seconds in total.

In Appendix I of the revised manuscript, we will specify the MILP solver, the relative optimality gap, and the stage-locking tolerances, and report the allocation overhead.

Table 5. Allocation overhead for LLaMA-3.1-8B with SparseGPT at a target layer sparsity of 50%.

Allocation StepTime (s)
Output gradient collection243.15
Candidate-state cost construction and other preparation3612.58
MILP solve time (Stage 1 / Stage 2 / Stage 3)0.637 / 0.835 / 0.683
Total runtime3857.89
Reviewer_4UFGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NS-UDPD to allocate different sparsity levels to different layers of large language models. It constructs a Diagonal-Kronecker NLL Sensitivity (DKNS) proxy cost by combining weight, input-activation, and output-NLL-gradient information to evaluate the pruning cost. It then introduces the Uniform Depth Prefix Dominance (UDPD) prefix constraint into joint discrete optimization to determine the layer-wise sparsity allocation. Experiments are conducted on the LLaMA-3 and Qwen3.5 series models with different fixed local pruning rules and 40%–70% sparsity. Results show that this framework improves model perplexity and downstream task performance; the paper also demonstrates its inference speedup at high sparsity when combined with SpInfer.

Critical strengths
  1. The paper has a clear and well-motivated methodological design. It introduces the DKNS pruning cost to capture the sensitivity of Language Modeling NLL to pruning, and identifies the limitation of minimizing only the total pruning cost, which may concentrate high costs in shallow layers and allow perturbations to propagate through subsequent layers. It then proposes UDPD, which constrains the cumulative pruning cost of each depth prefix relative to uniform sparsity allocation.
  2. The experimental results are convincing. The paper evaluates NS-UDPD across different sparsity levels, model architectures, and local pruning methods, demonstrating consistent improvements over existing layer-wise sparsity allocation methods. It also reports inference speedups when deploying the pruned models with an optimized sparse inference framework, providing additional evidence of its practical effectiveness.
Critical weaknesses
  1. The effectiveness of the proposed DKNS cost remains insufficiently validated. Although the authors provide a theoretical motivation for combining the three components, DKNS is still an approximation of the actual NLL degradation caused by pruning. It would be helpful to provide component-wise ablations and analyze the correlation between the estimated DKNS costs and the actual NLL changes after pruning.
  2. The motivation for using uniform sparsity allocation as the reference for UDPD constraints requires further validation. Since different layers exhibit varying pruning sensitivities, uniform allocation may not necessarily provide an appropriate reference for the depth-wise distribution of pruning costs. This design choice may limit the flexibility of the global sparsity allocation.
Questions for authors
  1. Could the authors provide an ablation study on the three components of DKNS, namely squared weights, input-activation second moments, and squared output-NLL-gradient energies? For example, how much does each component contribute to the final layer-wise sparsity allocation, and does combining all three consistently outperform using only one or two of them?
  2. The method uses uniform sparsity allocation to construct the depth-prefix cost bounds in UDPD. However, it is unclear why uniform allocation is a suitable reference, given the varying pruning sensitivities across layers. Could the authors provide further justification for this design choice and compare it with alternative reference strategies?
6 comments
AuthorsOpenReview ↗

Response to W1 and Q1 (Reviewer 4UFG)

Thank you for your constructive feedback and for recognizing the motivation behind NS-UDPD’s design and the strength of its experimental results. We respond to your comments point by point below.

Response to W1 and Q1 (Reviewer 4UFG)

Thank you for pointing out the need to assess the contribution of each DKNS factor and its relationship to actual NLL changes. Using Wanda on Qwen3.5-4B at a fixed target mean layer sparsity of 70%, we examine how each factor affects layer-wise sparsity allocation and post-pruning performance. We also analyze the rank correlation between DKNS state costs and actual NLL changes. We will add this controlled ablation results to Appendix G of the revised manuscript.

We evaluate all seven nonempty combinations of WW, AA, and GG, along with Uniform as a control. WW denotes the squared weight wℓ,m,i,j2w_{\ell,m,i,j}^{2}, AA the input-activation second moment A^ℓ,m,j\widehat{A}_{\ell,m,j}, and GG the squared output-NLL-gradient energy G^ℓ,m,i\widehat{G}_{\ell,m,i}. All seven combinations use the same calibration samples and follow the same Wanda rule for generating candidate deletion masks. For each combination, we set the omitted factors to 1, compute the uniform prefix bounds using the resulting costs, and run all three stages. Uniform assigns a target sparsity of 70% to every layer. Table 1 reports the results. ✓ indicates that the constraint is enabled or the corresponding stage is executed, and ✗ indicates that the constraint is disabled or the corresponding stage is skipped.

Under the same allocation procedure, adding GG to WAWA, AA to WGWG, and WW to AGAG reduces PPL from 107.34, 234.79, and 48.79, respectively, to 38.50 and increases six-task zero-shot mean accuracy by 4.55, 1.85, and 1.17 percentage points, respectively. In this setting, full DKNS outperforms all single-factor and two-factor variants on both metrics.

Table 1. DKNS factor ablation on Qwen3.5-4B with Wanda.

VariantUDPD Prefix ConstraintsStage 1Stage 2Stage 3WikiText-2 PPL ↓Six-Task Zero-Shot Mean (%) ↑
Uniform✗✗✗✗98.1837.20
W✓✓✓✓94.5937.26
A✓✓✓✓152.5736.18
G✓✓✓✓369.6338.02
WA✓✓✓✓107.3436.71
WG✓✓✓✓234.7939.41
AG✓✓✓✓48.7940.09
WAG✓✓✓✓38.5041.26

Correlation Between DKNS State Costs and actual NLL Changes. With Wanda, we compute the full-depth state costs of the same set of model-wide allocations using full DKNS (WAGWAG) and the activation-weighted cost (WAWA) obtained by setting G=1G=1. We also measure the change in full-model NLL for each allocation. For each cost variant, we pair the cost ranks with the ranks of the measured NLL changes and compute the Pearson correlation between the two sets of ranks to obtain Spearman's rank correlation coefficient ρ\rho ([−1,1][-1,1], with values nearer 1 indicating closer agreement). The values of ρ\rho for WAWA and WAGWAG are 0.214 and 0.976, respectively. Adding output-NLL-gradient weighting changes the rank correlation between DKNS state costs and actual NLL changes from weakly positive to strongly positive. In this setting, the state-cost ranking produced by full DKNS (WAGWAG) agrees more closely with the ranking of actual NLL changes than the ranking produced by WAWA.

Contribution of Each Factor to Layer-Wise Sparsity Allocation. We use Wanda on Qwen3.5-4B at a fixed target mean layer sparsity of 70%. Using the full WAGWAG allocation as the reference, we remove one factor at a time, solve the allocation problem again, and compare the resulting allocation with the reference layer by layer. We compute the absolute difference in sparsity between the two allocations for each layer, average these differences with equal weight across all 32 layers, and count how many layers change sparsity. Removing WW, AA, and GG changes the sparsity of 30, 31, and 32 layers, respectively, with mean absolute differences of 7.34, 12.58, and 9.92 percentage points. By mean absolute difference, the factors rank A>G>WA > G > W, and removing GG changes the sparsity of the most layers.

AuthorsOpenReview ↗

Response to W2 and Q2 (Reviewer 4UFG)

Response to W2 and Q2 (Reviewer 4UFG)

Thank you for asking why we use Uniform as the reference allocation and how this choice affects allocation flexibility.

Layer-Wise Pruning Sensitivity and the Uniform Reference Allocation. NS-UDPD uses the Uniform allocation at the same budget to bound the cumulative DKNS state cost separately at each depth prefix. Uniform sets every layer's sparsity to the target mean layer sparsity SS, so the budget directly determines the reference allocation. The uniform prefix bound UdU_d sums the rebased DKNS state costs of the first dd layers at sparsity SS. The increment Ud−Ud−1U_d-U_{d-1} equals the state cost of layer dd at SS. Thus, each layer's contribution to the bound reflects its pruning sensitivity as estimated by DKNS. Using Uniform as the reference lets us determine whether reallocating sparsity under the same budget increases the cumulative state cost at any depth prefix.

Flexibility of Global Sparsity Allocation. At every depth dd, allocation xx must satisfy Ad(x)≤UdA_d(x)\leq U_d, where Ad(x)A_d(x) and UdU_d are the cumulative DKNS state costs of the first dd layers under xx and Uniform at the same budget, respectively. Bounding each prefix prevents cost savings in deeper layers from offsetting excess costs in shallower prefixes. If the cumulative state cost of the first d−1d-1 layers is below its bound, the DKNS state cost of layer dd may exceed that layer's cost under Uniform by up to Ud−1−Ad−1(x)U_{d-1}-A_{d-1}(x). This slack leaves room for non-uniform allocation, allowing the joint optimization to adjust layer-wise sparsities while exactly meeting the budget and satisfying all prefix bounds.

We use SparseGPT on Qwen3.5-4B at a fixed target mean layer sparsity of 70%. All variants share the DKNS cost table and candidate grid. As shown in Table 2, Cost-Only minimizes only the full-depth DKNS state cost AL(x)A_L(x). UDPD (Uniform) constructs prefix bounds using 70% sparsity in every layer. ATP-shaped projects the ATP sparsity schedule, which increases with depth, onto the same grid while preserving the budget, then uses the projected sparsities to construct prefix bounds. DKNS-ranked reorders the same projected sparsities, assigning higher sparsities to layers with smaller DKNS cost increases when sparsity is raised by one grid step from SS. Each of the three reference-based variants jointly selects layer-wise sparsities under its own prefix bounds, sequentially minimizing the full-depth DKNS state cost, the worst normalized depth-prefix ratio, and the layer-wise sparsity variance.

At the same budget and with the same solver stages, UDPD (Uniform) reduces PPL by 22.12% relative to ATP-shaped and 18.27% relative to DKNS-ranked, while increasing six-task mean accuracy by 0.88 and 0.81 percentage points, respectively. ATP-shaped assigns lower reference sparsities to shallow layers, which may tighten the corresponding prefix bounds and limit how much pruning less sensitive shallow layers can accommodate. DKNS-ranked orders the entire set of layer sparsities using one-step DKNS cost increases at SS, and this ordering may vary across sparsity levels. These results support using Uniform as the default reference in this setting. We will include this analysis in Appendix H of the revised manuscript.

Table 2. Comparison of allocation strategies on Qwen3.5-4B with SparseGPT at a target mean layer sparsity of 70%. ✓ and ✗ indicate enabled and disabled, respectively.

Allocation StrategyPrefix ConstraintsStage 1Stage 2Stage 3WikiText-2 PPL ↓Six-Task Mean (%) ↑
Cost-Only✗✓✗✗29.2842.82
UDPD (Uniform)✓✓✓✓25.8543.54
ATP-shaped✓✓✓✓33.1942.66
DKNS-ranked✓✓✓✓31.6342.73

Dear Reviewer 4UFG,
Thank you again for your encouraging and insightful review. In our response, we addressed each of your points with detailed explanations and results. As the discussion phase closes on November 18, we would like to check whether these responses address all your concerns as you outlined, and we are happy to provide any further details. Thank you for your time!
Best,
Authors of 4857

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Summary of Reviews

We thank all reviewers for their careful evaluation and constructive feedback. The reviews recognized several aspects of the paper.

  1. The DKNS pruning cost is well motivated and captures the sensitivity of language-modeling negative log-likelihood (NLL) to pruning (Reviewer 4UFG).

  2. UDPD's depth-prefix constraints control how pruning costs accumulate across layers (Reviewers 4UFG, ggeW, and j7pF).

  3. UDPD retains uniform sparsity allocation as a feasible reference while allowing layer-wise sparsity to vary non-monotonically with depth (Reviewers ggeW and j7pF).

  4. The sparsity allocation formulation explicitly states the conditions for exact lexicographic optimization (Reviewer ggeW).

  5. NS-UDPD achieves strong perplexity results across models, sparsity levels, and local pruning rules (Reviewers 4UFG, ggeW, and j7pF).

Following the reviewers’ suggestions, we added and clarified:

  1. We ablated the individual DKNS factors and analyzed the rank correlation between DKNS state costs and actual changes in NLL.

  2. We compared full NS-UDPD with NS-CostOnly using identical DKNS state costs for both methods. NS-CostOnly minimizes only the full-depth DKNS state cost.

  3. We conducted a three-stage ablation with fixed DKNS state costs and UDPD depth-prefix constraints to assess the contributions of the two secondary objectives.

  4. We reported the actual parameter-weighted sparsity achieved by NS-UDPD on Qwen3.5's hybrid architecture and showed its performance advantage over the baselines.

  5. We reported the mixed-integer linear programming (MILP) solver settings and sparsity allocation overhead. We also conducted paired reruns across multiple calibration seeds.

The rebuttal and additional evidence addressed the principal concerns raised by the reviewers. We thank them for their careful assessment and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.