- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Residual-Aware Sparsification and Pruning (RASP), a training-free framework that aims to preserve dominant singular directions during LLM compression. Rather than directly masking the original weight matrix, RASP decomposes each projection into a principal low-rank component and a residual, applying pruning or activation sparsification only to the latter.
However, at its current stage, I have serious concerns about the submission, including: (1) the absence of the mandatory AI Use Statement required by ICLR 2027; (2) presentation and correctness concerns regarding Figures 1 and 2, including possible inconsistencies between the diagrams and the stated method (see Q5), as well as questions about their preparation and provenance; and (3) clear numerical and definitional inconsistencies in the experimental results (see Q3).
Despite these concerns, I have tried to provide constructive comments that may help the authors substantially improve the work in a future revision or submission.
Critical strengths
S1. Reasonably broad experimental coverage and an attempt at a unified framework.
The paper evaluates several backbone models and benchmark datasets, covering a reasonably broad range of model sizes and downstream tasks. Some results show meaningful improvements over the corresponding baselines, although I still have concerns about experimental reproducibility and fairness (Q4), and the improvements over the stronger TEAL baseline are sometimes limited.
I also appreciate the attempt to address both weight/channel pruning and activation sparsification within a unified framework (not a common setting in existing papers to my knowledge). Considering these two settings together potentially broadens the applicability of the proposed method. However, their distinct mechanisms, theoretical assumptions, and deployment implications should be differentiated more carefully throughout the paper (see Q2 and Q5).
Critical weaknesses
W1. Insufficient novelty and positioning against closely related research.
In my view, the central idea of RASP is not entirely new. Combining low-rank components with sparse residuals or using low-rank structure to compensate for pruning-induced errors has been investigated in several related works:
- OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition (ICLR 2025): training-free sparse-plus-low-rank weight approximation.
- SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression (ICML 2025): low-rank compensation for one-shot compression.
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025): joint sparse-plus-low-rank optimization.
- 1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models (EMNLP 2025 Findings): training-free integration of low-rank approximation and sparsification.
I recognize that these methods differ from RASP, particularly in its wrapper design and compatibility with activation-sparsity masks. Nevertheless, they substantially challenge the novelty and positioning of this work.
The current experiments mainly show improvements over existing pruning/sparsification baselines, but do not sufficiently isolate the contribution of principal-subspace protection from the general benefit of adding low-rank components. I would therefore expect a clearer account of the methodological novelty, supported by direct numerical comparisons under matched conditions, to establish why RASP is preferable to closely related alternatives. [Q1]
W2. Concerns about the theoretical analysis and its connection to the actual method. [Q2]
After carefully reading Section 3.3 and Appendix A, I have substantial concerns about the theoretical analysis. In its current form, I do not find the results sufficiently convincing to support the broader claims about principal-subspace preservation and practical model compression.
If theory is intended as a major contribution, I strongly encourage the authors to rigorously revisit the assumptions, scope of the guarantees, and their empirical validity. Specific concerns are detailed in Q2.1-Q2.3.
Additional suggestions for improving the paper (not separate critical weaknesses, in my personal view):
- Potential extension to attention modules. While I understand the focus on MLP projections, attention components may also exhibit meaningful directional and low-rank structure. For instance, Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024) investigates residual-stream communication subspaces, while CLOVER: Cross-Layer Orthogonal Vectors Pruning (ICML 2025) applies SVD to attention projection pairs. Exploring whether RASP extends to these components could provide additional insight.
- Clarifying the research motivation. The Abstract mentions sensitivity to the "scoring rule or sparsification criterion," suggesting a general-purpose wrapper design. I encourage the authors to better connect this motivation with the proposed mechanism and experimental validation.
- Presentation and organization. The distinction between pruning and activation sparsification is sometimes unclear, particularly in the theoretical and deployment discussions. I occasionally found it difficult to determine which setting a claim addressed. A clearer separation, or a more focused scope, would improve the presentation.
- Figure preparation and visual quality. Figures 1 and 2 deserve closer technical and visual verification. Please also clarify any generative AI assistance in their preparation, where applicable, following the conference's disclosure guidance. My primary concern is scientific accuracy and traceability.
- Minor details. Please check the inconsistent highlighting in Table 2 (the final HumanEval entry), small text in Figure 4, slight overflow in Table 7, and unresolved
Table ??references in the appendix.
Questions for authors
[Q1] Novelty and comparison with related sparse-plus-low-rank approaches.
As discussed in W1, could the authors clarify the essential differences between RASP and existing methods such as OATS, SLiM, 3BASiL, and SSLC? I would particularly appreciate direct numerical comparisons with closely related or matched-budget sparse-plus-low-rank baselines, together with a clearer explanation of what RASP contributes beyond existing decomposition and compensation strategies.
[Q2] Theoretical analysis and its validity.
[Q2.1] Does protecting the top singular directions necessarily preserve functionally important information?
The largest singular directions of a weight matrix are not necessarily the most important for downstream behavior, particularly when input activation distributions are considered. Why is weight-only truncated SVD the appropriate choice? Could the authors provide controlled comparisons with alternative or activation-aware subspace selection strategies to establish that preserving these directions is responsible for the improvements?
[Q2.2] How informative are the theoretical guarantees?
Lemma 1 establishes
\|W-\widetilde{W}_{r,M}\|_2\leq\sigma_{r+1}(W).$$ While correct for the linear operator, this bound follows directly from truncated SVD and the residual masking identity. Similarly, Theorem 1 provides a sufficient condition that already compares residual reconstruction error against a direct-pruning error lower bound. Could the authors clarify the **additional theoretical insight** these results provide? More importantly, how frequently do the sufficient conditions hold in actual LLM projections, and how well do they predict downstream performance improvements? **[Q2.3] Are the guarantees equally applicable to pruning and activation sparsification?** The paper unifies both settings through diagonal masking, but static weight/channel pruning and dynamic activation sparsification differ in their statistical properties and resource constraints. In particular, the equal-parameter-budget theorem concerns column pruning, whereas activation sparsity does not necessarily reduce stored parameters. The actual implementation also involves nonlinear gated MLPs. Could the authors **explicitly distinguish which guarantees apply to each setting** and clarify their applicability to nonlinear MLPs and practical inference efficiency? --- **[Q3] Serious experimental inconsistencies in Appendix B.6.** I found several inconsistencies between the experimental definitions, textual descriptions, and plotted results, which raise concerns about the reported rank-sparsity analysis. **[Q3.1] Inconsistent interpretation of residual sparsity.** Figure 8 is described as using **50% equivalent sparsity**, yet the authors state: > At $r=1024$, the residual has 0% sparsity, so the observed performance is effectively determined by $L_r$ alone. If residual sparsity is 0%, then $M=I$ and $$\widetilde{W}_{r,M}=L_r+R_r=W,$$ which recovers the dense projection rather than $L_r$ alone. If the authors instead mean 0% residual density (100% sparsity), this should be reconciled with the claimed 50% equivalent-sparsity budget. **Please clarify the actual experimental configuration, sparsity definition, and budget accounting.** **[Q3.2] Numerical inconsistencies between Figure 8 and the text.** Several reported PPL values do not match the corresponding plots: - **Figure 8(a), $r=1$:** plotted **16.95** vs. reported **19.95**. - **Figure 8(a), $r=256$:** plotted approximately **13.91** vs. reported **16.91**. - **Figure 8(b), RASP+Wanda:** best plotted PPL approximately **14.76** vs. reported **17.76**. These are not rounding differences and directly affect the interpretation of the rank-scaling results. I strongly request that the authors **verify Figure 8 against the original experimental logs**, explain any differences in experimental configurations, and conduct a broader numerical consistency check throughout the manuscript. --- **[Q4] Experimental fairness, reproducibility, and practical efficiency.** **[Q4.1] Wanda implementation and comparison fairness.** The original Wanda primarily performs weight-level pruning, whereas RASP uses channel-wise masks. How is Wanda's elementwise saliency adapted? Is the reported baseline the original implementation or a modified structured variant? This matters because the severe degradation of the baseline in some settings may depend on this adaptation, potentially affecting the fairness of the comparison. **[Q4.2] Fairness of 50% equivalent sparsity.** RASP introduces additional low-rank parameters and computation. Although equivalent sparsity accounts for nominal parameter counts, it does not necessarily ensure comparable memory traffic or computational costs across weight pruning and activation sparsification. Could the authors clarify **how equivalent sparsity is calculated and enforced across baselines**, including low-rank overhead, and provide more direct budget-matched comparisons? **[Q4.3] Limited scope of the reported 40%-46% speedup.** The reported speedups (Appendix B.1, Figure 6) come from **a single Llama 3 8B MLP projection on an H100**, rather than complete-model inference. Could the authors provide **end-to-end throughput, latency, and peak-memory measurements** under matched conditions? I am particularly interested in whether the modest accuracy gains over TEAL translate into meaningful improvements on the actual quality-efficiency frontier. --- **[Q5] Consistency and interpretation of Figure 2.** I would appreciate a careful verification of Figure 2 against the mathematical formulation and implementation. Several aspects appear unclear or potentially inconsistent. **(1) Dimensional consistency.** The intermediate projection outputs appear to be labeled as $\widetilde{W}_1$ and $\widetilde{W}_3$, which should denote matrices rather than vectors. Similarly, $$\hat{h}=\sigma(\widetilde{W}_1)\odot\widetilde{W}_3$$ appears to omit the input $x$. Should these instead be $\widetilde{W}_1x$ and $\widetilde{W}_3x$? **(2) Up/gate projection definitions.** Section 3.4 identifies $W_1$ as the up-projection and $W_3$ as the gate-projection, yet applies the activation to $W_1$. Conventional SwiGLU applies it to the gate projection. Are the labels reversed, or is a different convention used? Please confirm consistency with the implementation. **(3) Scope of drop-in compatibility.** RASP appears to require compatible channel-wise masks, while some existing pruning methods operate at different granularities. Could the authors clarify the actual scope of compatibility and any required adaptations? **(4) Are FFN channels actually removed?** The residual branches are masked, but the low-rank paths still contribute to all FFN channels: $$\hat{h} = \sigma\left((L_{1,r}+MR_{1,r})x\right) \odot \left((L_{3,r}+MR_{3,r})x\right).$$ Thus, masked channels may remain active through the protected low-rank components. Is RASP therefore more accurately described as **residual-computation sparsification rather than complete FFN-channel removal**? Please clarify the implications for the pruning claims and sparse-kernel implementation. --- **Overall assessment.** At this stage, I am inclined to recommend rejection, primarily due to concerns about **novelty, theoretical justification, and substantial experimental inconsistencies**. Nevertheless, I remain open to further discussion and hope these comments can help the authors identify and address the fundamental issues. I would particularly appreciate careful, evidence-based responses that could materially affect my assessment.