acceptodds
Withdrawn from ICLR 2027

Residual-Aware Sparsification and Pruning: Protecting Principal Subspaces During Sparsification or Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Linear layers dominate LLM parameter count and memory traffic, making them a major computational bottleneck. Existing structured pruning and activation-sparsity methods reduce these costs by removing or skipping channels according to saliency or activation-based criteria. However, such methods can discard dominant singular directions of the original projection, making performance sensitive to the scoring rule or sparsification criterion. We propose Residual-Aware Sparsification and Pruning (RASP), a training-free drop-in **wrapper** for pruning and sparsity methods. For each targeted weight \(\mW\), RASP decomposes \(\mW = \mL_r + \mR_r\), where \(\mL_r\) captures the top-\(r\) singular directions and \(\mR_r\) is the residual. RASP then confines pruning or sparsification to \(\mR_r\), preserving the dominant projection directions while bounding approximation error by the residual norm, rather than the full weight norm. At 50% sparsity, RASP consistently improves generation scores – by up to +7.33 points over GRIFFIN, +7.93 over Wanda, +2.10 over TEAL. Interestingly, RASP surpasses dense inference on several Gemma 2 9B IT generation benchmarks. Finally, RASP is able to achieve 40%-46% speed-ups compared to a dense model. Our code is publicly available at https://anonymous.4open.science/r/RASP-EB5E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Sparse Inference, Low-Rank Decomposition, Model Pruning, Fast Decoding, LLM Compression

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_hezTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Residual-Aware Sparsification and Pruning (RASP), a training-free framework that aims to preserve dominant singular directions during LLM compression. Rather than directly masking the original weight matrix, RASP decomposes each projection into a principal low-rank component and a residual, applying pruning or activation sparsification only to the latter.

However, at its current stage, I have serious concerns about the submission, including: (1) the absence of the mandatory AI Use Statement required by ICLR 2027; (2) presentation and correctness concerns regarding Figures 1 and 2, including possible inconsistencies between the diagrams and the stated method (see Q5), as well as questions about their preparation and provenance; and (3) clear numerical and definitional inconsistencies in the experimental results (see Q3).

Despite these concerns, I have tried to provide constructive comments that may help the authors substantially improve the work in a future revision or submission.

Critical strengths

S1. Reasonably broad experimental coverage and an attempt at a unified framework.

The paper evaluates several backbone models and benchmark datasets, covering a reasonably broad range of model sizes and downstream tasks. Some results show meaningful improvements over the corresponding baselines, although I still have concerns about experimental reproducibility and fairness (Q4), and the improvements over the stronger TEAL baseline are sometimes limited.

I also appreciate the attempt to address both weight/channel pruning and activation sparsification within a unified framework (not a common setting in existing papers to my knowledge). Considering these two settings together potentially broadens the applicability of the proposed method. However, their distinct mechanisms, theoretical assumptions, and deployment implications should be differentiated more carefully throughout the paper (see Q2 and Q5).

Critical weaknesses

W1. Insufficient novelty and positioning against closely related research.

In my view, the central idea of RASP is not entirely new. Combining low-rank components with sparse residuals or using low-rank structure to compensate for pruning-induced errors has been investigated in several related works:

I recognize that these methods differ from RASP, particularly in its wrapper design and compatibility with activation-sparsity masks. Nevertheless, they substantially challenge the novelty and positioning of this work.

The current experiments mainly show improvements over existing pruning/sparsification baselines, but do not sufficiently isolate the contribution of principal-subspace protection from the general benefit of adding low-rank components. I would therefore expect a clearer account of the methodological novelty, supported by direct numerical comparisons under matched conditions, to establish why RASP is preferable to closely related alternatives. [Q1]

W2. Concerns about the theoretical analysis and its connection to the actual method. [Q2]

After carefully reading Section 3.3 and Appendix A, I have substantial concerns about the theoretical analysis. In its current form, I do not find the results sufficiently convincing to support the broader claims about principal-subspace preservation and practical model compression.

If theory is intended as a major contribution, I strongly encourage the authors to rigorously revisit the assumptions, scope of the guarantees, and their empirical validity. Specific concerns are detailed in Q2.1-Q2.3.


Additional suggestions for improving the paper (not separate critical weaknesses, in my personal view):

  • Potential extension to attention modules. While I understand the focus on MLP projections, attention components may also exhibit meaningful directional and low-rank structure. For instance, Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024) investigates residual-stream communication subspaces, while CLOVER: Cross-Layer Orthogonal Vectors Pruning (ICML 2025) applies SVD to attention projection pairs. Exploring whether RASP extends to these components could provide additional insight.
  • Clarifying the research motivation. The Abstract mentions sensitivity to the "scoring rule or sparsification criterion," suggesting a general-purpose wrapper design. I encourage the authors to better connect this motivation with the proposed mechanism and experimental validation.
  • Presentation and organization. The distinction between pruning and activation sparsification is sometimes unclear, particularly in the theoretical and deployment discussions. I occasionally found it difficult to determine which setting a claim addressed. A clearer separation, or a more focused scope, would improve the presentation.
  • Figure preparation and visual quality. Figures 1 and 2 deserve closer technical and visual verification. Please also clarify any generative AI assistance in their preparation, where applicable, following the conference's disclosure guidance. My primary concern is scientific accuracy and traceability.
  • Minor details. Please check the inconsistent highlighting in Table 2 (the final HumanEval entry), small text in Figure 4, slight overflow in Table 7, and unresolved Table ?? references in the appendix.
Questions for authors

[Q1] Novelty and comparison with related sparse-plus-low-rank approaches.

As discussed in W1, could the authors clarify the essential differences between RASP and existing methods such as OATS, SLiM, 3BASiL, and SSLC? I would particularly appreciate direct numerical comparisons with closely related or matched-budget sparse-plus-low-rank baselines, together with a clearer explanation of what RASP contributes beyond existing decomposition and compensation strategies.

[Q2] Theoretical analysis and its validity.

[Q2.1] Does protecting the top singular directions necessarily preserve functionally important information?

The largest singular directions of a weight matrix are not necessarily the most important for downstream behavior, particularly when input activation distributions are considered. Why is weight-only truncated SVD the appropriate choice? Could the authors provide controlled comparisons with alternative or activation-aware subspace selection strategies to establish that preserving these directions is responsible for the improvements?

[Q2.2] How informative are the theoretical guarantees?

Lemma 1 establishes

\|W-\widetilde{W}_{r,M}\|_2\leq\sigma_{r+1}(W).$$ While correct for the linear operator, this bound follows directly from truncated SVD and the residual masking identity. Similarly, Theorem 1 provides a sufficient condition that already compares residual reconstruction error against a direct-pruning error lower bound. Could the authors clarify the **additional theoretical insight** these results provide? More importantly, how frequently do the sufficient conditions hold in actual LLM projections, and how well do they predict downstream performance improvements? **[Q2.3] Are the guarantees equally applicable to pruning and activation sparsification?** The paper unifies both settings through diagonal masking, but static weight/channel pruning and dynamic activation sparsification differ in their statistical properties and resource constraints. In particular, the equal-parameter-budget theorem concerns column pruning, whereas activation sparsity does not necessarily reduce stored parameters. The actual implementation also involves nonlinear gated MLPs. Could the authors **explicitly distinguish which guarantees apply to each setting** and clarify their applicability to nonlinear MLPs and practical inference efficiency? --- **[Q3] Serious experimental inconsistencies in Appendix B.6.** I found several inconsistencies between the experimental definitions, textual descriptions, and plotted results, which raise concerns about the reported rank-sparsity analysis. **[Q3.1] Inconsistent interpretation of residual sparsity.** Figure 8 is described as using **50% equivalent sparsity**, yet the authors state: > At $r=1024$, the residual has 0% sparsity, so the observed performance is effectively determined by $L_r$ alone. If residual sparsity is 0%, then $M=I$ and $$\widetilde{W}_{r,M}=L_r+R_r=W,$$ which recovers the dense projection rather than $L_r$ alone. If the authors instead mean 0% residual density (100% sparsity), this should be reconciled with the claimed 50% equivalent-sparsity budget. **Please clarify the actual experimental configuration, sparsity definition, and budget accounting.** **[Q3.2] Numerical inconsistencies between Figure 8 and the text.** Several reported PPL values do not match the corresponding plots: - **Figure 8(a), $r=1$:** plotted **16.95** vs. reported **19.95**. - **Figure 8(a), $r=256$:** plotted approximately **13.91** vs. reported **16.91**. - **Figure 8(b), RASP+Wanda:** best plotted PPL approximately **14.76** vs. reported **17.76**. These are not rounding differences and directly affect the interpretation of the rank-scaling results. I strongly request that the authors **verify Figure 8 against the original experimental logs**, explain any differences in experimental configurations, and conduct a broader numerical consistency check throughout the manuscript. --- **[Q4] Experimental fairness, reproducibility, and practical efficiency.** **[Q4.1] Wanda implementation and comparison fairness.** The original Wanda primarily performs weight-level pruning, whereas RASP uses channel-wise masks. How is Wanda's elementwise saliency adapted? Is the reported baseline the original implementation or a modified structured variant? This matters because the severe degradation of the baseline in some settings may depend on this adaptation, potentially affecting the fairness of the comparison. **[Q4.2] Fairness of 50% equivalent sparsity.** RASP introduces additional low-rank parameters and computation. Although equivalent sparsity accounts for nominal parameter counts, it does not necessarily ensure comparable memory traffic or computational costs across weight pruning and activation sparsification. Could the authors clarify **how equivalent sparsity is calculated and enforced across baselines**, including low-rank overhead, and provide more direct budget-matched comparisons? **[Q4.3] Limited scope of the reported 40%-46% speedup.** The reported speedups (Appendix B.1, Figure 6) come from **a single Llama 3 8B MLP projection on an H100**, rather than complete-model inference. Could the authors provide **end-to-end throughput, latency, and peak-memory measurements** under matched conditions? I am particularly interested in whether the modest accuracy gains over TEAL translate into meaningful improvements on the actual quality-efficiency frontier. --- **[Q5] Consistency and interpretation of Figure 2.** I would appreciate a careful verification of Figure 2 against the mathematical formulation and implementation. Several aspects appear unclear or potentially inconsistent. **(1) Dimensional consistency.** The intermediate projection outputs appear to be labeled as $\widetilde{W}_1$ and $\widetilde{W}_3$, which should denote matrices rather than vectors. Similarly, $$\hat{h}=\sigma(\widetilde{W}_1)\odot\widetilde{W}_3$$ appears to omit the input $x$. Should these instead be $\widetilde{W}_1x$ and $\widetilde{W}_3x$? **(2) Up/gate projection definitions.** Section 3.4 identifies $W_1$ as the up-projection and $W_3$ as the gate-projection, yet applies the activation to $W_1$. Conventional SwiGLU applies it to the gate projection. Are the labels reversed, or is a different convention used? Please confirm consistency with the implementation. **(3) Scope of drop-in compatibility.** RASP appears to require compatible channel-wise masks, while some existing pruning methods operate at different granularities. Could the authors clarify the actual scope of compatibility and any required adaptations? **(4) Are FFN channels actually removed?** The residual branches are masked, but the low-rank paths still contribute to all FFN channels: $$\hat{h} = \sigma\left((L_{1,r}+MR_{1,r})x\right) \odot \left((L_{3,r}+MR_{3,r})x\right).$$ Thus, masked channels may remain active through the protected low-rank components. Is RASP therefore more accurately described as **residual-computation sparsification rather than complete FFN-channel removal**? Please clarify the implications for the pruning claims and sparse-kernel implementation. --- **Overall assessment.** At this stage, I am inclined to recommend rejection, primarily due to concerns about **novelty, theoretical justification, and substantial experimental inconsistencies**. Nevertheless, I remain open to further discussion and hope these comments can help the authors identify and address the fundamental issues. I would particularly appreciate careful, evidence-based responses that could materially affect my assessment.
Reviewer_m5rKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work studies training-free pruning of LLM linear layers. It is motivated by the observation that standard channel pruning can significantly damage the dominant singular directions of linear layers. The proposed method, RASP, wraps existing pruning methods. It decomposes a weight matrix into the best rank-r approximation and the residual via truncated SVD and applies the wrapped pruning method only to the residual. The paper gives error bounds showing that the linear approximation error is bounded by the residual norm and empirically shows that RASP can improve GRIFFIN, Wanda, and TEAL on Llama, Gemma, and DeepSeek-distilled Qwen models.

Critical strengths
  • The writing is good for the most part (see weakness 1). The core idea is made understandable quickly and the methods section is easy to follow. (The score of 1 in presentation is mostly due to Fig. 2)
  • The selection of models and benchmarks the proposed method is tested on is extensive enough to obtain meaningful results.
  • The authors supply Triton kernels and show evidence that compute demands of a forward pass are reduced compared to the base methods.
Critical weaknesses
  • The existing literature was not reviewed adequately. Not comparing against, let alone cite, essentially identical approaches is inacceptible for a top tier ML conference submission. Also, every experimental comparison is base method vs. base method + RASP. There are no comparisons against other low rank + sparse methods.
    • R-Sparse (ICLR 2025) [1]: Equivalent to TEAL+RASP. Not cited. Should be a baseline.
    • OATS (ICLR 2025) [2]: Same low rank + pruned residual structure but jointly optimized. Not cited. Should be a baseline.
    • LoSparse (ICML 2023) [3]: Same parameterization as the Wanda and GRIFFIN variants but used during fine-tuning. Not cited.
    • SVDQuant (ICLR 2025) [4]: Same "protect the principal subspace, compress the residual" approach but for quantization. Not cited.
  • The paper overclaims substantially in some parts. For example, the "surpasses dense inference" on Gemma-2-9B-IT and the "substantial gain" on AIME24 are too strong. The gains are too small to be claimed for greedy decoding/single seed runs on these benchmarks.
    • MATH500 (Table 5, Gemma-2-9B): Dense sits at 25.20 and TEAL+RASP is at 27.20. That is 10 more correct answers out of 500 with greedy decoding.
    • AIME24 (Table 4, DeepSeek-R1-Distill-Qwen-7B): TEAL is at 36.67 and TEAL+RASP is at 43.33. Two more correct answers out of 30 from one sampled run at temperature 0.6.
  • One of the main pillars of the experimental design, "equivalent sparsity", is never defined properly. Sec. 3.3 is very hand-wavy about it.
  • The derived error bound is correct, but trivial and not very useful, since it does not depend on the mask applied to the residual.
  • The hand-drawn style of Fig. 2 is not acceptible for a top-tier ML conference. The fact that the white backgroud of Fig. 1 does not match the white background of the document, while less problematic, is still not acceptible.

[1] Zhang, Zhenyu, et al. "R-sparse: Rank-aware activation sparsity for efficient llm inference." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Zhang, Stephen, and Vardan Papyan. "OATS: Outlier-aware pruning through sparse and low rank decomposition." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Li, Yixiao, et al. "Losparse: Structured compression of large language models based on low-rank and sparse approximation." International Conference on Machine Learning. PMLR, 2023.

[4] Li, Muyang, et al. "Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

No questions

Reviewer_hXkiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RASP, a training-free add-on for existing pruning and activation-sparsity methods in LLMs. RASP uses a truncated SVD to split each MLP weight matrix into a dense low-rank part and a residual. The base method’s mask is applied only to the residual, so the main directions of the weight are always kept.

Critical strengths
  1. The idea is simple and general. RASP does not change how the base method scores channels, only changes where the mask is applied, so it can wrap different pruning and activation-sparsity methods.
  2. The gains are large for aggressive pruners.
Critical weaknesses
  1. My main concern is about the novelty compared to R-sparse [1]. The paper does not cite or discuss R-Sparse, which is very similar. R-Sparse keeps the large input channels with the original weights and sends the remaining channels through a low-rank path built from an offline SVD of the same weights. This is the same computation as RASP’s equivalent form in Sec. 3.6, with the same factorization. TEAL+RASP therefore closely resembles R-Sparse with TEAL’s thresholds and a fixed rank of 256. The main new part seems to be extending this decomposition to channel-pruning methods such as GRIFFIN and WANDA. The authors should cite R-Sparse, compare with it under the same budget, and clarify what is new relative to it.
  2. Limited baselines are tested. Only the three base methods are evaluated. Related methods discussed in Sec. 4 are not compared experimentally, such as CAP, which also uses a low-rank + sparse decomposition, WINA and CATS.

[1] Zhang Z, Liu Z, Tian Y, Khaitan H, Wang Z, Li S. R-sparse: Rank-aware activation sparsity for efficient llm inference. InInternational Conference on Learning Representations 2025 May 1 (Vol. 2025, pp. 77452-77466).

Questions for authors
  1. Could the authors compare TEAL+RASP with R-Sparse under the same budget, in both accuracy and latency, and clarify what RASP adds beyond R-Sparse?
  2. How are WANDA’s per-weight scores turned into the diagonal channel mask M that RASP requires? Table 1 lists WANDA as not using a structured mask.
  3. For TEAL+RASP, which dimension does the mask act on?
Reviewer_eHggOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free add-on method for activation-sparsity methods. It decomposes the weight into a SVD part and a residual part, and only apply mask to the residual part. The evaluation on various models prove its effectiveness.

Critical strengths
  1. The propose method is training-free and simple to apply to various method.
  2. The experiments cover many models and tasks, and the gains are promising.
Critical weaknesses
  1. The decomposition method seems similar to the R-Sparse paper (https://arxiv.org/abs/2504.19449), but it is not cited or discussed in the paper.
  2. All results seem come from a single run
Questions for authors
  1. How is RASP different from R-Sparse? Please add a comparison.
  2. What is the end-to-end decoding speed (tokens/s) compared with dense and TEAL?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.