- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes K-Support Pruning (KSP), a framework for post-training N:M pruning of LLMs. It uses blockwise, activation-weighted squared k-support norms to formulate a convex regularized layer reconstruction problem, followed by mask hardening and a masked least-squares refit. The theoretical analysis provides a weighted proximal operator, a conditional lossless-hardening certificate, and support recovery under diagonal curvature. Extending the norm to fractional budgets yields a water-level interpretation for allocating sparsity across matrices, with discrete patterns selected under parameter or latency budgets. Evaluation spans seven models across four architecture families, plus a Llama-3.1-70B deployment check, and includes uniform sparsity comparisons, component ablations, and measured latency experiments. KSP generally achieves competitive or improved quality, while the ablations highlight the substantial contributions of refitting and budget allocation.
Critical strengths
- The paper provides a unified framework for N:M pruning and sparsity allocation through the k-support norm. The weighted proximal operator and water-level interpretation make the framework computationally constructive.
- Experiments cover multiple model families and sparsity levels, with shared-refit and latency-matched controls. These comparisons demonstrate competitive pruning quality and the practical value of compensation and budget allocation.
Critical weaknesses
- The theoretical guarantees are narrower than some claims suggest: tightness applies to the scaled per-block sparse ball, while exact recovery requires diagonal curvature. The paper also does not report how often the lossless-hardening certificate holds in its LLM experiments.
- The comparisons do not fully isolate the benefit of the proposed regularizer under matched computation and compensation. For example, on Llama-3.2-3B, KSP improves perplexity over ALPS from 18.20 to 17.93 but takes 2200 versus 1570 seconds, while the shared-refit ablation excludes ALPS and SparseGPT.
Questions for authors
-
Clarification of novelty: Which parts of the variational representation and proximal algorithm are new beyond the box-norm framework of McDonald et al. (2016) [1], which already provides an algorithm for the squared-norm prox? Please distinguish the weighted extension and pruning-specific results from inherited machinery.
-
Evidence connecting theory to practice: What fraction of blocks satisfy the hardening certificate at the actual returned iterates? Please report the associated hardening loss and convergence residuals, and clarify which guarantees apply to the finite-budget, validation-selected ADMM output.
-
Evidence for attribution: Under matched optimization time and the same final masked-CG refit, how does KSP compare with ALPS and SparseGPT on representative 2:4 and 75%-sparsity settings? Does the advantage persist across calibration seeds?
-
Clarification of allocation: Which weights are used to compute , and is the jointly convex weight–budget program actually solved, or is allocation computed with fixed weights before pruning? This distinction is important for interpreting the allocation optimality claim and its relationship to the deployed procedure.
[1] McDonald A M, Pontil M, Stamos D. New perspectives on k-support and cluster norms[J]. Journal of Machine Learning Research, 2016, 17(155): 1-38.
Response to Reviewer x8VM
Thank you for the precise questions; each is answered below and in the revised PDF.
W1 — Scope of the guarantees. We agree, and now state the scope where each claim is made: tightness is per block, for the ℓ2-bounded N-sparse set (abstract, Prop. 1); exact recovery holds under diagonal curvature with a separation condition (abstract, §3.3); and the hardening result is a per-block condition (Cor. 1), whose measured frequency is reported under Q2.
Q1 — Novelty beyond McDonald et al. (2016). Delineated next to Thm. 1. Inherited: the variational box-norm form, the unweighted squared-norm prox with its O(d log d) algorithm, and non-integer budgets (Argyriou et al. 2012; McDonald et al. 2016). New: identifying this geometry as the tight per-block relaxation of N:M sparsity, and the pruning-specific results it yields — the prox with separate loss and penalty weights and its selection statistic ρ_j; matched scaling (the weighting, unique up to scale, under which ρ_j ranks by the loss); the lossless-hardening condition; support recovery under diagonal curvature; and the zero-iteration limit that recovers magnitude and Wanda.
Q2 — Certificate frequency, hardening loss, residuals, and what holds for the returned output (Tabs. 23–24). The reported solver (Alg. 3, App. C) is a splitting with a ridge variable B, a masked variable D given by exact blockwise support selection (the hardened, debiased counterpart of the prox step) and a dual V; B is the variable hardened. Diagnostics at the returned iterate (seed 0; residuals are medians over pruned matrices, median 33 iterations):
W2 / Q3 — Same refit, matched time, seeds (Tab. 2, Tabs. 20–22, Fig. 10). Every mask now receives the identical masked-CG refit from W0; WikiText-2, mean ± sd over calibration seeds:
Q4 — Allocation: which weights, and is the joint program solved? (App. E.3, Alg. 5). The allocation is computed once, with fixed weights, before the final pruning pass, and then held fixed; the final prune restarts from W0. The deployed allocator is a greedy descent on the latency-priced knapsack of §4, each step valued by the measured KL change of the whole model with previously committed matrices in place (probe sequences from the C4 and WikiText-2 training splits, disjoint from all evaluation data). Thm. 2 characterizes the separable surrogate G_T, evaluated on the dense weights W0; in our tests the measured value performed better, so it is the one deployed. The jointly convex weight–budget program of App. D is a theoretical object and is not solved in the pipeline; the revision states the deployed procedure explicitly (Alg. 5).