- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper deals with structured pruning in a setting where any (re)training is excluded. After demonstrating that pruning importance scores share a common quadratic form, this paper proposes a lower-triangle basis change to achieve a pruning subspace in which node removal is possible while compensating for the loss on the remaining nodes via LLS (Linear Least Squares). The proposed SNP (Subspace Node Pruning) method appears to outperform SAW (Summed Absolute Weights) and PFA-EN (Principal Filter Analysis) for VGG16 on ImageNet, and Intra-Fusion for ResNet-50 on ImageNet. Furthermore, on WikiText, SNP appears to achieve lower perplexity compared to LLM Surgeon for OPT-125M and LLaMA-2 7B.
Critical strengths
The idea is elegant and unifying. The paper introduces a common quadratic framework for the importance scores (activation, Taylor, Fisher), where pruning and least-squares reconstruction form a single operation, with importance measures that are exactly additive within a layer.
Critical weaknesses
The extension beyond linear layers is not specified: and the LDL are defined per unit, but no details are provided for groups (k² patches, attention heads) or for the residuals, even though the abstract mentions the heads and layers.
The experimental evaluation is limited.
First, the evaluation of LLMs relies exclusively on WikiText-2 perplexity. As it has been demonstrated in recent literature on language model compression (notably [1]), stable perplexity can mask a loss of capabilities. No zero-shot or few-shot application benchmarks (for example MMLU or GSM8K) are reported to demonstrate that the pruned models remain usable.
Second, recent post-training baselines are missing. For langage models, the paper omits direct comparisons with key methods that do not require retraining, such as Wanda ([2], which is cited in Table 1 but not evaluated), or its structured variant Wanda-sp, and OPTIN [3]. For vision: the evaluation on ResNet-50 is restricted to ‘Group 6’ alone (Figure 3) when compared with Intra-Fusion, without reporting the overall accuracy on ImageNet. Furthermore, no standard framework for managing structural dependencies (e.g. DepGraph [4]) is compared, and Vision Transformers (ViT or Swin) are completely absent from the experiments, despite a dedicated paragraph in the introduction.
Absence of actual physical compression in LLMs: The authors specify (lines 334-336) that for the dimensions shared with the residual flow and between the Q, K, and V projections, the channels are not removed but simply set to zero. Consequently, the reported sparsity does not correspond to a reduction in the size of the hidden states or to a gain in memory bandwidth during sequential inference, thereby negating a large portion of the practical benefits of structured pruning.
Unsupported claims regarding computational cost: The authors conclude that their method is ‘cheaper to compute’ (lines 467-470) than sequential methods with compensation. However, no actual execution time (wall-clock time), no measure of maximum memory usage, nor any count of operations for the pruning phase are provided. Inverting the correlation matrix and recalculating the LDL decomposition across 40 shots using K-FAC backward passes represents a significant cost that needs to be quantified.
References:
[1] Jaiswal et al.,"Compressing LLMs: The Truth is Rarely Pure and Never Simple", ICLR 2024
[2] Sun et al., "A Simple and Effective Pruning Approach for Large Language Mode", ICLR 2024
[3] Khaki et al., "The need for speed: Pruning transformers with one recipe", ICLR24
[4] Fang et al., "Depgraph: Towards any structural pruning", CVPR 2023
[5] Men et al., "Shortgpt: Layers in large language models are more redundant than you expect", ACL 2025
[6] Yang et al., "Laco: Large language model pruning via layer collapse.", EMNLP 2024
Questions for authors
-
Practical evaluation of LLMs. How does SNP actually perform on standardised reasoning and knowledge tasks (e.g. MMLU or GSM8K) using LLaMA-2 7B and OPT-125M, beyond the WikiText-2 perplexity? Furthermore, cross-referencing Table 2, Table 4, and Table 5 reveals that single-shot SliceGPT performs on-par with 1-shot SNP at 10% and 20% sparsity (e.g., 5.96 vs. 5.89 on LLaMA-2 7B, and 34.53 vs. 34.95 on OPT-125M), and even clearly outperforms 1-shot SNP at 30% sparsity (8.64 vs. 19.14 on LLaMA-2 7B, and 44.61 vs. 55.13 on OPT-125M). Could the authors clarify their positioning against SliceGPT? Outperforming SliceGPT across every single sparsity is not strictly necessary. However, I believe it is important to understand what distinct practical or architectural advantages SNP brings to the table.
-
Hardware gains and actual latency. Given that the residual flow and attention head channels are set to zero rather than physically removed, what are the actual gains in VRAM, power consumption or inference time? ShortGPT [5] and LaCo [6] show that removing or merging around 25% of the layers often preserves functional performance far better than width pruning without retraining. Even if the granularity differs (depth vs. width), why implement a complex LDL decomposition if removing entire blocks offers a better latency-to-accuracy trade-off?
-
Computation time of SNP. What is the actual computation time (wall-clock time) required by SNP compared with LLM Surgeon (for 1, 5 and 40 shots) and compared with fast post-training approaches such as Wanda or SliceGPT? If the computation budget of SNP is high enough, can one consider apply a basic importance score with a small retraining phase corresponding to the same budget ?
-
Overall evaluation on ResNet-50. Why is the comparison in Figure 3 limited to ‘Group 6’? Does that mean that pruning was only performed in Group 6 ? Does the FLOP measure reported in x-axis correspond to only parameters in Group 6 or to all the parameters in the network ? What is the overall Top-1 accuracy achieved on ImageNet for the entire ResNet-50 architecture when it is pruned end-to-end?
-
Absence of Vision Transformers. The introduction includes a specific discussion of pruning in Transformers (lines 83-92). Why was SNP not evaluated on vision architectures such as ViT, Swin or DeiT?
Things to improve the paper that did not impact the score:
Clarity of Algorithm 1.
The code is provided but I did not find a readme to guide the execution, some architectures are loadable with the options but not discussed in the paper (e.g. ResNet-101, VGG-11 or VGG-19).
Figure 4 lack tick marks on x-axis, Figure 5 lacks xaxis legend and tick marks (maybe an oversight since Figure 6 has it and whole figure got split in half?).
Given the strengths and weaknesses mentioned above, my current assessment leans toward a clear rejection. That said, I would be willing to reconsider my position if the authors can provide convincing evidence that the concerns above are resolved.
Withdrawal and response to the reviewers
We would like to thank all three reviewers for their careful reading of our work and for the detailed feedback. After considering the reviews, we have decided to withdraw the submission in order to ensure that we can take as much time as is necessary to address all of the concerns and requests raised by the reviewers.
We also, however, wanted to take this opportunity to share some reflections and clarifications on our work and this review that might be of general interest.
Contribution: Reviewers 9z6G and ymNF both noted that, for a fixed order of removal, our latent importances coincide with the costs that greedy sequential methods such as OBC and ZipLM compute through repeated compensation. This is correct. All of these methods, alongside Chee et al. (2022), LLM Surgeon, and the least-squares approaches of Mariet & Sra and He et al., reconstruct pruned units with the same least-squares step (our Eq. 3). They differ only in the importance measure, together with the unit order, and in how per-layer pruning ratios are set. Greedy iterative methods score units one at a time. Each score therefore already reflects the units that remain available for reconstruction. Applied to these methods, SNP changes neither their scores nor their ordering. Non-iterative methods, such as LLM Surgeon, K-OBD, and magnitude- or Taylor-based scores, instead score all units at once. Their predicted cost of pruning a group of correlated units is then inconsistent with their own decision to prune many units in a single step, and this distorts the predicted impact of pruning. SNP keeps the local order of such methods fixed and corrects their scores, and with them the per-layer ratios, from a single factorization. Choosing the order itself is a combinatorial problem. Our contribution in this case can therefore be clarified as: a cheap, single-operation correction of the scores of non-iterative methods, which removes one measurable source of error and leaves the remaining error attributable to the stated assumptions (the ordering, the importance measure, and interactions across layers). A direct comparison of OBC, ZipLM and Chee et al. against SNP applied to non-iterative methods then compares these true remaining errors. We shall make these comparisons in the future.
Loss-optimality of the compensation. Reviewer 9z6G raised the concern that Eq. 3 is a least-squares solution in activation space, and therefore not loss-optimal for loss-based measures. We believe this point is worth clarifying, as it holds more generally than the paper made apparent. Under the K-FAC approximation, the loss change caused by a weight change ΔW is ΔL(ΔW)=21tr(GΔWCΔW⊤), where G is the Gram matrix of top-down gradients and C is the input Gram matrix. Pruning the input units P while keeping K fixes ΔW:P=−W:P and leaves ΔW:K free. The update of Eq. 3, ΔW:K∗=W:PCPKCKK−1, satisfies [ΔW∗C]:K=0. Any other feasible update can be written as ΔW∗+E with E:P=0, and its loss exceeds that of ΔW∗ by exactly 21tr(GECE⊤)≥0. Eq. 3 is therefore the loss-optimal compensation for any G, not only for activation-based measures. We agree with the reviewers that additivity is exact only within a layer, and that interactions across layers remain an approximation.
The collapse of LLM Surgeon at one shot. Reviewer 9z6G asked whether the 1-shot LLM Surgeon results reflect a configuration issue. Our experiments used the LLM Surgeon codebase with its original hyperparameters, changing only the number of shots, and SNP was run through the same pipeline. The failure of LLM Surgeon is therefore a genuine collapse of the model rather than a bug. Our interpretation is that when per-unit costs are computed independently, there is an assumption that correlated neighbours will compensate for that unit's removal. However, if those same neighbours are removed at the same time, i.e. in the same shot, then the true loss far exceeds the predicted loss (as in our Figure 4). Many-shot schedules hide this error by re-estimating costs between shots. This is precisely the error that SNP is designed to fix.