acceptodds
Withdrawn from ICLR 2027

Subspace Node Pruning: Retraining-Free Neural Network Compression via Orthogonal Subspace Projections

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Node pruning is the art of removing computational units such as neurons, filters, attention heads, or even entire layers to significantly reduce inference time while retaining network performance. In this work, we show that a broad family of importance measures, from activation-based to Fisher-based, share a common quadratic form, and that through a lower-triangular change of basis we can reach a pruning subspace in which whole nodes can be pruned with a simultaneous recovery of the impact of lost units via linear least squares. Furthermore, the importance measures within our subspace are decoupled and additive through an orthogonalization of the layer activations, allowing multi-node pruning within layers without interference between importance scores. Our method, Subspace Node Pruning (SNP), outperforms the compared baselines on VGG-16 and ResNet-50 on ImageNet. On OPT-125M and LLaMA-2 7B, it matches the state-of-the-art LLM Surgeon method at a 40-shot schedule and outperforms it in every case where fewer shots are used. All of this is achieved without any retraining after pruning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
pruning, node pruning, retraining-free, orthogonal subspaces

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_s4t9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper deals with structured pruning in a setting where any (re)training is excluded. After demonstrating that pruning importance scores share a common quadratic form, this paper proposes a lower-triangle basis change to achieve a pruning subspace in which node removal is possible while compensating for the loss on the remaining nodes via LLS (Linear Least Squares). The proposed SNP (Subspace Node Pruning) method appears to outperform SAW (Summed Absolute Weights) and PFA-EN (Principal Filter Analysis) for VGG16 on ImageNet, and Intra-Fusion for ResNet-50 on ImageNet. Furthermore, on WikiText, SNP appears to achieve lower perplexity compared to LLM Surgeon for OPT-125M and LLaMA-2 7B.

Critical strengths

The idea is elegant and unifying. The paper introduces a common quadratic framework for the importance scores (activation, Taylor, Fisher), where pruning and least-squares reconstruction form a single operation, with importance measures that are exactly additive within a layer.

Critical weaknesses

The extension beyond linear layers is not specified: Il,i⊥I_{l,i}^\perp and the LDL are defined per unit, but no details are provided for groups (k² patches, attention heads) or for the residuals, even though the abstract mentions the heads and layers.

The experimental evaluation is limited.
First, the evaluation of LLMs relies exclusively on WikiText-2 perplexity. As it has been demonstrated in recent literature on language model compression (notably [1]), stable perplexity can mask a loss of capabilities. No zero-shot or few-shot application benchmarks (for example MMLU or GSM8K) are reported to demonstrate that the pruned models remain usable.
Second, recent post-training baselines are missing. For langage models, the paper omits direct comparisons with key methods that do not require retraining, such as Wanda ([2], which is cited in Table 1 but not evaluated), or its structured variant Wanda-sp, and OPTIN [3]. For vision: the evaluation on ResNet-50 is restricted to ‘Group 6’ alone (Figure 3) when compared with Intra-Fusion, without reporting the overall accuracy on ImageNet. Furthermore, no standard framework for managing structural dependencies (e.g. DepGraph [4]) is compared, and Vision Transformers (ViT or Swin) are completely absent from the experiments, despite a dedicated paragraph in the introduction.

Absence of actual physical compression in LLMs: The authors specify (lines 334-336) that for the dimensions shared with the residual flow and between the Q, K, and V projections, the channels are not removed but simply set to zero. Consequently, the reported sparsity does not correspond to a reduction in the size of the hidden states or to a gain in memory bandwidth during sequential inference, thereby negating a large portion of the practical benefits of structured pruning.

Unsupported claims regarding computational cost: The authors conclude that their method is ‘cheaper to compute’ (lines 467-470) than sequential methods with compensation. However, no actual execution time (wall-clock time), no measure of maximum memory usage, nor any count of operations for the pruning phase are provided. Inverting the correlation matrix ClC_l and recalculating the LDL decomposition across 40 shots using K-FAC backward passes represents a significant cost that needs to be quantified.

References:
[1] Jaiswal et al.,"Compressing LLMs: The Truth is Rarely Pure and Never Simple", ICLR 2024
[2] Sun et al., "A Simple and Effective Pruning Approach for Large Language Mode", ICLR 2024
[3] Khaki et al., "The need for speed: Pruning transformers with one recipe", ICLR24
[4] Fang et al., "Depgraph: Towards any structural pruning", CVPR 2023
[5] Men et al., "Shortgpt: Layers in large language models are more redundant than you expect", ACL 2025
[6] Yang et al., "Laco: Large language model pruning via layer collapse.", EMNLP 2024

Questions for authors
  1. Practical evaluation of LLMs. How does SNP actually perform on standardised reasoning and knowledge tasks (e.g. MMLU or GSM8K) using LLaMA-2 7B and OPT-125M, beyond the WikiText-2 perplexity? Furthermore, cross-referencing Table 2, Table 4, and Table 5 reveals that single-shot SliceGPT performs on-par with 1-shot SNP at 10% and 20% sparsity (e.g., 5.96 vs. 5.89 on LLaMA-2 7B, and 34.53 vs. 34.95 on OPT-125M), and even clearly outperforms 1-shot SNP at 30% sparsity (8.64 vs. 19.14 on LLaMA-2 7B, and 44.61 vs. 55.13 on OPT-125M). Could the authors clarify their positioning against SliceGPT? Outperforming SliceGPT across every single sparsity is not strictly necessary. However, I believe it is important to understand what distinct practical or architectural advantages SNP brings to the table.

  2. Hardware gains and actual latency. Given that the residual flow and attention head channels are set to zero rather than physically removed, what are the actual gains in VRAM, power consumption or inference time? ShortGPT [5] and LaCo [6] show that removing or merging around 25% of the layers often preserves functional performance far better than width pruning without retraining. Even if the granularity differs (depth vs. width), why implement a complex LDL decomposition if removing entire blocks offers a better latency-to-accuracy trade-off?

  3. Computation time of SNP. What is the actual computation time (wall-clock time) required by SNP compared with LLM Surgeon (for 1, 5 and 40 shots) and compared with fast post-training approaches such as Wanda or SliceGPT? If the computation budget of SNP is high enough, can one consider apply a basic importance score with a small retraining phase corresponding to the same budget ?

  4. Overall evaluation on ResNet-50. Why is the comparison in Figure 3 limited to ‘Group 6’? Does that mean that pruning was only performed in Group 6 ? Does the FLOP measure reported in x-axis correspond to only parameters in Group 6 or to all the parameters in the network ? What is the overall Top-1 accuracy achieved on ImageNet for the entire ResNet-50 architecture when it is pruned end-to-end?

  5. Absence of Vision Transformers. The introduction includes a specific discussion of pruning in Transformers (lines 83-92). Why was SNP not evaluated on vision architectures such as ViT, Swin or DeiT?

Things to improve the paper that did not impact the score:
Clarity of Algorithm 1.
The code is provided but I did not find a readme to guide the execution, some architectures are loadable with the options but not discussed in the paper (e.g. ResNet-101, VGG-11 or VGG-19).
Figure 4 lack tick marks on x-axis, Figure 5 lacks xaxis legend and tick marks (maybe an oversight since Figure 6 has it and whole figure got split in half?).

Given the strengths and weaknesses mentioned above, my current assessment leans toward a clear rejection. That said, I would be willing to reconsider my position if the authors can provide convincing evidence that the concerns above are resolved.

Reviewer_ymNFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce Subspace Node Pruning (SNP), which is a retraining free structured pruning method by change pruning to a lower triangular basis. The main idea is to construct an ordered latent basis where the activation correlation is removed. The resulting transformation is obtained through a LDL factorization / Gram-Schmidt-type orthogonalization. The weight update is shown to coincide with a linear least square reconstruction of the removed units from the retained ones.

SNP is evaluated on VGG-16 and ResNet-50, and OPT-125M and LLaMA-2 7B. The strongest results appear in the low-shot LLM pruning regime, where SNP substantially outperforms LLM Surgeon, while the gap narrows when many pruning shots are used.

Critical strengths

The main conceptual contribution is clear. The paper does not merely introduce another heuristic importance score, it instead reformulates multi-unit pruning in a basis where correlation between units is explicitly removed. This gives a clean interpretation of why pruning costs can become additive within a layer.

The empirical behavior in the low-shot regime is interesting and consistent with the paper's motivation. On OPT-125M, SNP is substantially more robust than LLM Surgeon when many units are pruned in a small number of shots, while the two methods become much closer with a 40-shot schedule.

Critical weaknesses

The novelty relative to sequential compensated pruning is not yet sufficiently clear. At a high level, SNP appears closely related with quantities that sequential OBS/OBC-style methods would obtain through repeated compensation and updates. If this interpretation is correct, then the main contribution is not a fundamentally different pruning objective, but rather an efficient factorized reformulation of sequential compensated pruning.

The experimental comparison is not sufficiently comprehensive. I would like to see a broader comparison with recent retraining-free structured pruning methods, particularly methods designed for one-shot or few-shot structured pruning. And I would like to see a broader comparison with recent retraining-free structured pruning methods, particularly methods designed for one-shot or few-shot structured pruning.

Questions for authors
  1. For a fixed pruning order, is SNP exactly equivalent to performing sequential optimal-compensation pruning and collecting all sequential costs in advance? If so, can the authors state and prove this formally?

  2. What is the complexity of SNP compared with LLM Surgeon, OBC/OBS-style sequential pruning, and ZipLM, including factorization and re-estimation costs?

  3. How much wall-clock pruning time is actually saved in the OPT-125M and LLaMA-2 7B experiments?

  4. Can the authors report variance over different calibration subsets or random seeds? Some of the gains in the many-shot regime are relatively small.

Reviewer_9z6GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that activation-, first-order-, and Fisher/K-FAC-based importance scores share a quadratic form tr(S_l C_l). It applies a lower-triangular (LDL / Gram–Schmidt) change of basis to the layer activations, so latent importances become exactly additive within a layer. Truncating that basis is equivalent to removing original units with least-squares compensation. Units are ordered by an OBS-style score, S_ii / [C^{-1}]_ii. Without retraining, SNP beats simple baselines on VGG-16 and Intra-Fusion on ResNet-50. On OPT-125M and LLaMA-2 7B it outperforms LLM Surgeon mainly when few pruning shots are used.

Critical strengths
  1. The unified quadratic view of importance measures is clean, and the lower-triangular argument (Fig. 1) gives a clear account of why PCA/ZCA bases do not remove units while LDL does.

  2. Merging pruning and least-squares reconstruction into a single operation, with additive within-layer costs, is elegant and cheap.

  3. The few-shot result against LLM Surgeon is a useful practical finding (Table 4: SNP is better in 59 of 60 OPT-125M settings). Fig. 4 and Figs. 5–6 back the claim that per-unit costs underestimate joint pruning cost.

Critical weaknesses
  1. Limited novelty. For a fixed order, the latent importances equal greedy sequential OBS costs (the authors acknowledge this for OBC/ZipLM), and the ordering score is the OBS removal cost used in LLM Surgeon. Least-squares reconstruction (Mariet & Sra; He et al.) and pivoted-QR unit selection (Chee et al., 2022) are closely related. The contribution is mainly a reformulation, and a direct comparison to Chee et al., OBC, and ZipLM is missing.

  2. "Exact" additivity depends on approximations. It holds only under a sample-shared sensitivity S_l (K-FAC style) and only within a layer. Also, the compensation in Eq. 3 is least-squares in activation space (C_l), not loss-optimal under S_l for the loss-based measures.

  3. Overclaimed results. The abstract says SNP beats LLM Surgeon "in every case where fewer shots are used", but Table 2 shows LLaMA-2 7B at 5 shots and 10% sparsity with LS 5.27 vs SNP 5.49. LLM Surgeon's 1-shot numbers (e.g., 7237 perplexity at 10%) look like a degenerate failure that needs explanation. At 40 shots the gains are small (48.89 vs 49.72; 14.88 vs 15.38), with no variance reported.

  4. Thin experiments. Vision baselines are weak (SAW, PFA-EN), and pruning still loses a lot of accuracy (about 40% Top-1 at 50% FLOPs on VGG-16). There is no ViT result. LLM evaluation is WikiText-2 perplexity only, with no zero-shot tasks, and LLaMA-2 7B has many missing entries.

  5. No real efficiency evidence. The LLM pruning zeroes parameters rather than removing them, and no wall-clock latency or pruning-time cost (LDL/curvature) is reported, even though efficiency is a central claim.

Questions for authors
  1. How does SNP differ, in theory and empirically, from Chee et al. (2022) pivoted QR + least squares, and from OBC/ZipLM, at equal compute?

  2. Why does LLM Surgeon collapse at 1 shot (7237 perplexity at 10% on LLaMA-2 7B)? Is this a configuration issue?

  3. What are the wall-clock pruning time and the actual inference speedup (with units physically removed) compared with LLM Surgeon?

  4. For loss-based measures, have you tried compensation weighted by S_l instead of plain least squares on C_l?

  5. Can you add zero-shot downstream accuracy for LLaMA-2 7B, plus a ViT result?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Withdrawal and response to the reviewers

We would like to thank all three reviewers for their careful reading of our work and for the detailed feedback. After considering the reviews, we have decided to withdraw the submission in order to ensure that we can take as much time as is necessary to address all of the concerns and requests raised by the reviewers.

We also, however, wanted to take this opportunity to share some reflections and clarifications on our work and this review that might be of general interest.

Contribution: Reviewers 9z6G and ymNF both noted that, for a fixed order of removal, our latent importances coincide with the costs that greedy sequential methods such as OBC and ZipLM compute through repeated compensation. This is correct. All of these methods, alongside Chee et al. (2022), LLM Surgeon, and the least-squares approaches of Mariet & Sra and He et al., reconstruct pruned units with the same least-squares step (our Eq. 3). They differ only in the importance measure, together with the unit order, and in how per-layer pruning ratios are set. Greedy iterative methods score units one at a time. Each score therefore already reflects the units that remain available for reconstruction. Applied to these methods, SNP changes neither their scores nor their ordering. Non-iterative methods, such as LLM Surgeon, K-OBD, and magnitude- or Taylor-based scores, instead score all units at once. Their predicted cost of pruning a group of correlated units is then inconsistent with their own decision to prune many units in a single step, and this distorts the predicted impact of pruning. SNP keeps the local order of such methods fixed and corrects their scores, and with them the per-layer ratios, from a single factorization. Choosing the order itself is a combinatorial problem. Our contribution in this case can therefore be clarified as: a cheap, single-operation correction of the scores of non-iterative methods, which removes one measurable source of error and leaves the remaining error attributable to the stated assumptions (the ordering, the importance measure, and interactions across layers). A direct comparison of OBC, ZipLM and Chee et al. against SNP applied to non-iterative methods then compares these true remaining errors. We shall make these comparisons in the future.

Loss-optimality of the compensation. Reviewer 9z6G raised the concern that Eq. 3 is a least-squares solution in activation space, and therefore not loss-optimal for loss-based measures. We believe this point is worth clarifying, as it holds more generally than the paper made apparent. Under the K-FAC approximation, the loss change caused by a weight change ΔW\Delta W is ΔL(ΔW)=12tr(GΔWCΔW⊤)\Delta\mathcal{L}(\Delta W) = \tfrac{1}{2}\mathrm{tr}(G \Delta W C \Delta W^\top), where GG is the Gram matrix of top-down gradients and CC is the input Gram matrix. Pruning the input units PP while keeping KK fixes ΔW:P=−W:P\Delta W_{:P} = -W_{:P} and leaves ΔW:K\Delta W_{:K} free. The update of Eq. 3, ΔW:K∗=W:PCPKCKK−1\Delta W^{\ast} _ {:K} = W _ {:P} C_{PK} C_{KK}^{-1}, satisfies [ΔW∗C]:K=0[\Delta W^{\ast} C]_ {:K} = 0. Any other feasible update can be written as ΔW∗+E\Delta W^{\ast} + E with E:P=0E_{:P} = 0, and its loss exceeds that of ΔW∗\Delta W^{\ast} by exactly 12tr(GECE⊤)≥0\tfrac{1}{2}\mathrm{tr}(G E C E^\top) \geq 0. Eq. 3 is therefore the loss-optimal compensation for any GG, not only for activation-based measures. We agree with the reviewers that additivity is exact only within a layer, and that interactions across layers remain an approximation.

The collapse of LLM Surgeon at one shot. Reviewer 9z6G asked whether the 1-shot LLM Surgeon results reflect a configuration issue. Our experiments used the LLM Surgeon codebase with its original hyperparameters, changing only the number of shots, and SNP was run through the same pipeline. The failure of LLM Surgeon is therefore a genuine collapse of the model rather than a bug. Our interpretation is that when per-unit costs are computed independently, there is an assumption that correlated neighbours will compensate for that unit's removal. However, if those same neighbours are removed at the same time, i.e. in the same shot, then the true loss far exceeds the predicted loss (as in our Figure 4). Many-shot schedules hide this error by re-estimating costs between shots. This is precisely the error that SNP is designed to fix.

Overclaims and missing evidence. Reviewer 9z6G correctly identified that the abstract's claim of outperforming LLM Surgeon "in every case where fewer shots are used" is contradicted by LLaMA-2 7B at 5 shots and 10% sparsity. More importantly, computational efficiency was central to our argument and yet we reported no wall-clock times. We have since measured these on OPT-125M. A single shot takes approximately 77 s with LLM Surgeon and 84 s with SNP, and the saving therefore comes entirely from needing fewer shots. To reach LLM Surgeon's best perplexity on OPT-125M, SNP requires 2 rather than 30 shots at 10% sparsity, 25 rather than 40 at 20% and 30%, and 15 rather than 40 at 50%, but the same number at 40%. The resulting speedups range from none to roughly an order of magnitude, which is a more modest and more variable picture than the paper suggested. We also agree with Reviewer s4t9 that zeroing, rather than removing, channels shared with the residual stream and across the Q, K and V projections forgoes much of the practical benefit of structured pruning. A convincing case requires the comparison with local importance methods that are used in combination with physical removal, with measured latency and memory.

Structured groups. Reviewer s4t9 noted that the paper does not specify how SNP handles convolutional filters, attention heads, or other structurally coupled units. SNP inherits this choice from the importance measure it corrects. For convolutions, we unfold the input into patches of dimension k2cink^2 c_\text{in}, and the parameters belonging to a single filter are adjacent in the fixed local order. Since importances are exactly additive in our subspace, the corrected importance of a filter is simply the sum of the latent importances of its parameters. Attention heads, and any other units that the importance measure treats as structurally coupled, are handled in the same way. This should have been stated explicitly in the paper.

We are grateful to the reviewers and the area chair for their time and effort, which have substantially improved our perspective on how to present and place this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.