acceptodds
Under review as a conference paper at ICLR 2027

NMP-SVD: Rethinking Dense Projections via N:M-Structured Low-Rank Compression for LLMs

Abstract

Existing large language model (LLM) compression methods based on Singular Value Decomposition (SVD) mainly focus on weighted reconstruction, singular-value truncation, and rank allocation, while largely leaving the representation of retained projection directions unexplored. Most existing methods preserve these projections as dense factors. We revisit this assumption through controlled perturbation experiments and find that low-rank representations can tolerate substantial directional approximation, with expanded approximate projections often outperforming smaller exact dense representations. Motivated by this observation, we propose NMP-SVD ( Projection SVD), a plug-and-play structured-sparsity enhancement for existing SVD compressors. NMP-SVD directly optimizes both low-rank factors under the original weighted reconstruction objective using dual-factor -OBS calibration, expands projection directions from the residual of the actual sparse solution, and applies block-wise backfitting to correct sequential expansion errors. A per-linear-layer selector further retains dense realizations for sparsity-sensitive layers. NMP-SVD preserves the weighting and rank-allocation strategies of the underlying compressor and is therefore compatible with diverse SVD frameworks. Experiments across multiple models and datasets demonstrate improved quality–compression trade-offs, higher prefill efficiency, and reduced memory usage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Singular Value Decomposition, Low-Rank Compression, N:M Structured Sparsity, Large Language Models, Sparse Projection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2vgmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes NMP-SVD, a plug-and-play enhancement for SVD-based LLM compression that uses N:M-structured sparse low-rank factors to retain more projection directions under a limited budget. It combines dual-factor calibration, residual-based projection expansion, and block-wise backfitting, while retaining dense factors for sparsity-sensitive layers. Experiments across multiple models and datasets show improved quality–compression trade-offs, higher prefill efficiency, and lower memory usage.

Critical strengths
  1. The method combines low-rank decomposition with N:M structured sparsity to retain more projection directions under a limited storage budget, while reducing reconstruction error through alternating dual-factor calibration and residual-guided expansion.
  2. The experiments cover multiple model families, scales, compression ratios, and SVD backbones, providing broad evidence of the method’s effectiveness and compatibility. Component ablations and end-to-end latency and memory measurements further support its design choices and practical benefits.
Critical weaknesses
  1. The decoding speedup is limited to high-concurrency settings in the reported experiments, while slower decoding at smaller batch sizes and the absence of batch-one evaluation leave its benefits for interactive deployment unclear.
  2. The layer-wise selector minimizes local reconstruction error without explicitly accounting for interactions across compressed layers. Consequently, these local choices may not improve the model’s final output distribution.
Questions for authors
  1. How much additional compression time and peak memory does NMP-SVD require relative to the underlying SVD compressor, particularly for the 32B and 70B models?
  2. Can author report decoding latency at smaller batch sizes, particularly batch size one, and across different context lengths to clarify the benefits for interactive deployment?
  3. How does adaptive selection compare with the all-sparse variant under matched storage budgets, and what fraction of layers retain dense representations?
  4. Could you compare alternative calibration strategies to better isolate the contribution of the proposed alternating dual-factor calibration?
6 comments
AuthorsOpenReview ↗

Response to Reviewer, Part 1/4

Q1: Compression Time and Memory Overhead.

We thank the reviewer for raising this question. NMP-SVD introduces additional offline computation for alternating factor calibration, residual-guided expansion, and backfitting. The total compression times and peak memory are already reported in Appendix D (Table 8). We summarize the results for Qwen3-32B and LLaMA2-70B below.

Total compression time under uniform rank allocation at a 40% base parameter-reduction ratio. NMP-SVD uses 2:4 sparsity and ρ=1.5\rho=1.5. Times are in hours/minutes; peak memory (GiB) follows the slash.

MethodQwen3-32BLLaMA2-70B
ASVD29 min / 7.35 GiB1 h 42 min / 10.47 GiB
+NMP-SVD3 h 11 min / 21.59 GiB5 h 22 min / 27.67 GiB
SVD-LLM1 h 11 min / 31.78 GiB6 h 55 min / 39.87 GiB
+NMP-SVD4 h 01 min / 31.78 GiB11 h 55 min / 39.87 GiB
KFAC-SVD2 h 05 min / 23.46 GiB8 h 43 min / 30.49 GiB
+NMP-SVD8 h 11 min / 23.46 GiB16 h 01 min / 30.50 GiB

For Qwen3-32B and LLaMA2-70B, the peak GPU memory remains nearly unchanged when integrated with SVD-LLM and KFAC-SVD, while ASVD exhibits a larger increase. Nevertheless, the peak memory remains manageable under layer-wise compression. Importantly, these additional optimization costs are incurred only once during offline compression and do not introduce additional computation at inference time.

AuthorsOpenReview ↗

Response to Reviewer, Part 2/4

W1/Q2: Small-Batch Decoding Efficiency.

We thank the reviewer for highlighting this limitation. The original decoding experiments employed general-purpose N:M sparse kernels, whose coarse-grained GEMM tiles are inefficient for the narrow matrix shapes encountered at small batch sizes, resulting in low GPU utilization and relatively high execution overhead.

To address this limitation, we developed two dedicated GPU-efficient sparse decoding kernels:

(1) Fine-Grained SIMT Kernel (B=1B=1–44). We replace coarse-grained Tensor Core GEMM with fine-grained, vector-oriented sparse computation (GEMV-style) using CUDA-core SIMT parallelism. This better matches small-batch workloads.

(2) Small-N CUTLASS Kernel (B=8B=8–128128). We retain 2:4 Sparse Tensor Core computation but reduce the GEMM tile size from 128×128×64128\times128\times64 to 64×64×6464\times64\times64, improving parallelism and reducing tile underutilization.

We further evaluated NMP-SVD with these optimized kernels at a 40% base compression ratio and ρ=1.4\rho=1.4, varying both batch size and KV context length. The results are summarized below.

Decoding efficiency across batch sizes (KV length = 2048). Each entry reports latency (ms) / peak memory (GiB).

BatchDenseDense-SVDNMP-SVDSpeedup vs. Dense-SVD
124.13 / 15.2227.94 / 10.0624.58 / 9.131.137$\times$
224.12 / 15.4829.03 / 10.3224.63 / 9.391.179$\times$
424.46 / 15.9929.00 / 10.8328.80 / 9.901.007$\times$
826.08 / 17.0129.32 / 11.8527.72 / 10.261.058$\times$
1629.00 / 19.0628.98 / 13.9027.69 / 12.311.047$\times$
3244.87 / 23.1538.81 / 17.9938.32 / 16.401.013$\times$

Decoding efficiency across KV context lengths (B=1B=1). Each entry reports latency (ms) / peak memory (GiB).

KV LengthDenseDense-SVDNMP-SVDSpeedup vs. Dense-SVD
25624.24 / 15.0028.00 / 9.8424.66 / 8.911.136$\times$
51224.13 / 15.0327.95 / 9.8724.63 / 8.941.135$\times$
102424.18 / 15.1027.98 / 9.9424.59 / 9.011.138$\times$
204824.13 / 15.2227.94 / 10.0624.58 / 9.131.137$\times$

With the optimized kernels, NMP-SVD achieves lower decoding latency and peak memory than Dense-SVD across all evaluated settings. At B=1B=1 and KV length 2048, latency decreases from 27.94 to 24.58 ms (1.137$\times$ speedup), while peak memory decreases from 10.06 to 9.13 GiB. The speedup reaches 1.179$\times$ at B=2B=2 and remains approximately 1.14$\times$ across KV lengths of 256–2048 at B=1B=1. Notably, at B=1B=1, NMP-SVD also approaches the latency of the original dense model while reducing peak memory from 15.22 to 9.13 GiB.

These results demonstrate that dedicated sparse execution kernels can overcome the small-batch inefficiency of generic sparse GEMM implementations, extending the practical benefits of NMP-SVD to low-concurrency decoding. We have updated Section 5.5 to describe the optimized kernels and analyze the additional decoding results.

AuthorsOpenReview ↗

Response to Reviewer, Part 3/4

W2: Why Local Adaptive Selection Remains Useful Without Global Guarantees.

We thank the reviewer for this comment. NMP-SVD is designed as a plug-and-play enhancement to existing SVD compressors, preserving their original weighting and rank allocation strategies (e.g., SVD-LLM-v2, KFAC-LEMS, and Zero-Sum SVD). Our goal is to improve how the retained projections are represented, rather than introducing a new global compression strategy.

We agree that lower local reconstruction error does not guarantee better final model outputs, because errors may interact and propagate across compressed layers. However, adaptive selection is intended as a simple safeguard rather than a global optimizer. Since sparsity tolerance varies across layers, we retain the original Dense-SVD representation whenever the expanded sparse realization produces higher local weighted reconstruction error. This prevents the selection of a locally worse representation under the inherited reconstruction objective, although it does not guarantee better downstream performance.

Importantly, in our evaluated configurations, the sparse realization requires less storage than its dense counterpart, including sparsity metadata. Thus, adaptive selection never increases storage beyond the original Dense-SVD baseline, while allowing sparsity-tolerant layers to benefit from more approximate projections. This provides a practical safeguard without changing the underlying SVD compression framework.

Q3: Adaptive Selection vs. All-Sparse and Dense Layer Ratios

We thank the reviewer for this suggestion. NMP-SVD does not rely on adaptive selection for its effectiveness, as demonstrated by the all-sparse results in Table 1. We further compare all-sparse and adaptive selection under the same base ranks and expansion ratios in Table 4.

Comparison of all-sparse and adaptive selection (KFAC-SVD, ρ=1.4).

ModelReductionVariantWT2 ↓AVG ↑Dense LayersDense Ratio
LLaMA2-13B20%All-sparse6.3858.710/2800%
Adaptive6.2859.26115/28041.07%
LLaMA2-13B40%All-sparse10.0648.260/2800%
Adaptive9.8849.3099/28035.36%
LLaMA3-8B20%All-sparse12.4953.710/2240%
Adaptive12.1854.20118/22452.68%
LLaMA3-8B40%All-sparse50.6737.100/2240%
Adaptive51.2136.5486/22438.39%

Adaptive selection improves both perplexity and average accuracy in three of the four configurations, although improvements in local reconstruction error do not always translate into better downstream performance.

The fraction of retained dense layers also varies across models and compression levels. For LLaMA2-13B, the dense ratio decreases from 41.07% to 35.36% as base reduction increases from 20% to 40%, while LLaMA3-8B decreases from 52.68% to 38.39%. Appendix C further provides detailed layer-wise selection patterns across different SVD backbones and expansion ratios, demonstrating heterogeneous sparsity tolerance.

We note that these comparisons use identical base ranks and expansion ratios, rather than strictly matched final storage budgets. Nevertheless, adaptive selection remains below the original Dense-SVD storage budget in the evaluated configurations.

Reviewer_rHg8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

SVD-based compression methods reduce the memory and inference latency of large models by representing weight matrices as a multiplication of two low rank matrices. Likewise, N:M sparsity tackles these issues by enforcing a structured sparsity pattern within weight matrices that specialized hardware can take advantage of. In this work, the authors explore decomposing LLM weight matrices into low rank matrices that also exhibit N:M sparsity. The authors validate the proposed method by combining it with many existing SVD based compression methods. At similar effective parameter amounts, the authors show that the proposed method consistently achieves better performance that SVD-based compression alone.

Critical strengths
  • The combination of N:M sparsity and SVD-based compression is novel and seems to provide advantages over SVD compression alone.
Critical weaknesses
  • There is a discrepancy in the numbers presented in tables 3 and 4 and the numbers in the main result tables. For example, table 4 states that the proposed method achieves a wikitext-2 perplexity of 12.18 and average accuracy of 54.20 for Llama-3 8B at 20% parameter reduction. The corresponding row in Table 1 shows a WikiText perplexity of 12.49 and an average accuracy of 53.71. More concerning, at 40% parameter reduction the results in the main table seem to correspond to the results for the “all sparse” variant rather than the proposed method which uses adaptive sparsity. This raises concerns that the results in the main table represent whichever variant achieves better performance.
  • A compelling point of SVD-based compression methods is that they provide a reduction in memory and compute without needing custom kernels or specialized hardware. N:M sparsity, on the other hand, only provides acceleration benefits for hardware able to take advantage of it. I think it is necessary to additionally compare with N:M sparsity methods as they are also able to advantage of the specialized hardware necessary for the proposed method.
Questions for authors
  • Can the authors clarify the descrepancies between Tables 3 and 4?
  • Can the authors provide comparisons between the proposed method and N:M sparsity pruning methods?
3 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for recognizing the novelty of our work and for the constructive comments. We address the two concerns below.

Q1/W1: Clarification of the discrepancies between Tables 1, 3, and 4.

We thank the reviewer for the constructive comments.

We would like to clarify that Tables 1 and 3 intentionally disable adaptive dense–sparse selection to isolate the effectiveness of sparse projection realization, while Table 4 separately evaluates the contribution of adaptive selection under the same KFAC-SVD backbone and expansion ratio (ρ=1.4\rho=1.4).

Importantly, our main experiments in Table 2 employ the full NMP-SVD framework with adaptive dense–sparse selection, evaluating its effectiveness on recent advanced SVD compressors, including SVD-LLM-v2, KFAC-LEMS, and Zero-Sum SVD.

Specifically, for LLaMA3-8B at 20% compression, Table 1 reports a WikiText2 perplexity of 12.49 and an average accuracy of 53.71%, matching the All Sparse results in Table 4, rather than the adaptive results (12.18 and 54.20%). Similarly, at 40% compression, Table 1 reports the All Sparse results (50.67 and 37.10%), rather than the adaptive results (51.21 and 36.54%).

Thus, Table 1 consistently follows a fixed experimental protocol rather than selectively reporting the better-performing variant. Since adaptive selection is based on local weighted reconstruction error rather than downstream evaluation metrics, it does not necessarily improve end-to-end performance in every setting. We have also revised the corresponding descriptions in the main text to make these experimental distinctions more explicit and avoid potential misunderstandings.

Q2/W2: Comparison with direct N:M weight pruning methods.

We thank the reviewer for this important suggestion. We agree that hardware-agnostic acceleration is an advantage of conventional Dense-SVD. However, NMP-SVD explores a different trade-off within SVD-based compression. Rather than preserving all projections as dense factors, we use hardware-supported N:M sparsity to represent more approximate projection directions under a constrained storage budget. We acknowledge that realizing sparse computational acceleration requires compatible hardware.

Since both NMP-SVD and direct N:M weight pruning can leverage the same sparse hardware, we additionally compare them under comparable parameter budgets. The base SVD rank is selected such that the expanded 2:4 sparse factors retain approximately the same number of nonzero parameters as direct 2:4 weight pruning.

Talbe: Comparison with direct 2:4 weight pruning. NMP-SVD uses ASVD-Uniform with 50% compression ratio and ρ=2\rho=2.

ModelMethodWT2 ↓\downarrowAVG ↑\uparrow
LLaMA2-13BWanda (weight 2:4)8.3257.00
SparseGPT (weight 2:4)7.0157.95
NMP-SVD (factor 2:4)8.1859.23

The results show trade-offs. NMP-SVD achieves the highest average accuracy on LLaMA2-13B.

Although both approaches benefit from N:M-capable hardware, they exploit different forms of redundancy. Direct N:M pruning sparsifies the original weights, whereas NMP-SVD builds upon low-rank decomposition and uses structured sparsity to increase projection capacity under a fixed storage budget. For models whose weight matrices are well suited to low-rank approximation, NMP-SVD can therefore offer a better quality–compression trade-off than direct pruning methods such as SparseGPT. Our goal is not to universally outperform direct N:M pruning, but to provide an alternative projection realization that improves existing SVD frameworks and offers advantages when low-rank structure can be effectively exploited.

Importantly, NMP-SVD achieves higher prefill efficiency than Dense-SVD and also delivers decoding speedups over Dense-SVD with our optimized sparse decoding kernels (Section 5.5), including at small batch sizes in our evaluated settings.

Thank you for your response and for the honest treatment of the N:M sparsity comparison. I also appreciate the clarification and now understand that tables 1 and 3 explicitly do not use adaptive selection. However, as adaptive selection is presented as a contribution of the method, the main experimental setup should not disable it. In general, adaptive selection seems to be insufficiently evaluated to be presented as a core contribution. In addition, I feel that the method section and overall presentation needs to be heavily revised. In particular, I found section 4.3 unapproachable on my initial read of the paper. Finally, although the additional N:M sparsity comparisons are helpful, I feel that the paper needs a more comprehensive evaluation with these methods than what has been provided during the rebuttal. For these reasons, I am not able to recommend acceptance and therefore keep my score.

AuthorsOpenReview ↗

Clarification of Core Contributions, Adaptive Selection, and Experimental Design

We thank the reviewer for the further feedback. We would like to clarify the distinction between our core methodological contributions and adaptive selection.

The core contribution of NMP-SVD lies in N:M-structured low-rank projection optimization, combining dual-factor N:M-OBS calibration, sparse-residual-guided projection expansion, and block-wise backfitting to improve the quality–compression trade-off. Adaptive selection is not essential to these components; rather, it serves as a simple, optional safeguard that retains the original Dense-SVD solution for sparsity-sensitive layers, facilitating robust plug-and-play integration with existing SVD compressors.

Accordingly, Tables 1 and 3 deliberately disable adaptive selection to isolate and evaluate our core contributions. Table 4 separately examines its impact on downstream performance, while the Appendix further analyzes its layer-wise selection behavior across different models, SVD backbones, and expansion ratios. Importantly, our main full-framework evaluation in Table 2 already incorporates adaptive selection, demonstrating NMP-SVD's compatibility with advanced SVD compressors, including SVD-LLM-v2, KFAC-LEMS, and Zero-Sum SVD.

We also acknowledge the concerns regarding presentation. We will revise Section 4.3 to clarify the algorithmic workflow, particularly the relationship between sparse calibration, residual-guided expansion, and block-wise backfitting.

Reviewer_RAd1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes NMP-SVD. The authors empirically demonstrate that sparsification + SVD can enhance the rank size compared with simply dense SVD and thus design low-rank factors as N:M-structured sparse projections. Experimental results show that NMP-SVD achieves a better compression performance on several LLM families compared with other SVD baseline methods.

Critical strengths
  • The paper identifies a very interesting phenomenon that sparsification + SVD can enhance the rank size compared with simply dense SVD, thus achieving a better compression performance.
  • NMP-SVD is better than other baseline methods according to the experimental results.
Critical weaknesses

(1) Although the phenomenon is interesting, most of the technical methods incorporated in NMP-SVD, including N:M sparsity, OBS-style calibration, residual low-rank expansion, and adaptive layer selection are common and NMP-SVD seems just an integration of them. (2) The evaluation benchmark is too simple. It is unclear whether NMP-SVD is also good on some real-world challenging tasks such as reasoning, coding, instruction following, and long-context generation. (3) According to the inference speedup experiment, NMP-SVD is slower than Dense-SVD for small decoding batches, which weakens the claim of broadly improved inference efficiency.

Questions for authors

(1) Could the authors clarify what part of the proposed optimization is fundamentally different from directly applying existing structured pruning techniques to low-rank factors? (2) Could the authors provide additional evaluations on more challenging downstream tasks, such as mathematical reasoning, code generation, instruction following, or long-context benchmarks? (3) Could the authors evaluate the inference efficiency of NMP-SVD under more realistic serving settings, especially smaller decoding batch sizes and longer context lengths, and provide more explanations on why it is slower than Dense-SVD?

3 comments
AuthorsOpenReview ↗

Response to Reviewer, Part 1/3

We thank the reviewer for recognizing projection realization as a useful and distinct design dimension in SVD-based LLM compression. And thanks the questions about methodological novelty, generation-intensive capabilities, and practical inference efficiency. We address each below with the existing ablations and new measurements.

Q1/W1: Fundamental difference between NMP-SVD and directly applying existing structured pruning techniques to low-rank factors.

We agree that N:M sparsity, OBS-style calibration, and residual expansion are established techniques. However, our contribution goes beyond their direct combination, with two key insights and a corresponding optimization method:

(1) Projection fidelity–capacity trade-off. Existing SVD compressors primarily focus on accurately preserving a limited number of dense projection directions. Our controlled experiments in Section 3 reveal that more approximate projections can outperform fewer highly accurate dense projections under a constrained storage budget. This insight is not restricted to N:M sparsity, which we adopt as a practical, hardware-supported realization.

(2) Direct sparsification is insufficient for weighted SVD. Simply applying N:M pruning to expanded SVD factors can cause severe performance degradation. The underlying challenge is that low-rank reconstruction, projection sparsity, and the original SVD weighting must be considered jointly. In particular, as discussed in Section 4.1, methods such as KFAC-SVD employ input- and output-side weighting. Imposing N:M sparsity in the transformed weighted space generally destroys the sparse structure when mapping back to the original coordinates. Moreover, independently pruning the two factors does not directly minimize their joint weighted reconstruction error.

(3) Factor-aware joint optimization. To address these challenges, we formulate N:M-constrained projection realization directly under the original weighted reconstruction objective. Rather than independently applying conventional OBS to the two factors, we incorporate the original two-sided weighting and the current opposite factor into the OBS curvature, constructing a factor-conditioned, alternating dual-factor solver. We further integrate sparse-residual-guided progressive expansion and backfitting to jointly refine the sparse low-rank representation.

The distinction can be summarized as follows:

Direct factor pruning:

(AS,BS)=(PN:M(AD),PN:M(BD))(A_S,B_S)=(P_{N:M}(A_D),P_{N:M}(B_D))

NMP-SVD objective:

\left\\|G^{1/2}(W-A_SB_S)H^{1/2}\right\\|_F^2$$ **Additional experimental evidence.** We further evaluated direct N:M pruning using Magnitude, Wanda, and SparseGPT on expanded low-rank factors, with the results summarized below. **Table: directly applying N:M pruning versus NMP-SVD.** | Method | Pruning description | Qwen3-8B WT2 ↓ | Qwen3-8B AVG ↑ | LLaMA2-13B WT2 ↓ | LLaMA2-13B AVG ↑ | |:---:|:---:|:---:|:---:|:---:|:---:| | Original | Uncompressed reference model | 9.71 | 69.86 | 4.88 | 67.19 | | KFAC-SVD | dense factors at base rank $r$ | 23.23 | 42.02 | 11.58 | 46.10 | | +Magnitude-N:M | directly pruning of factors | 42066.38 | 32.94 | 691.76 | 32.94 | | +Wanda-N:M | directly pruning of factors | 406.86 | 35.51 | 107.38 | 33.31 | | +SparseGPT-N:M | directly pruning of factors | 157.96 | 35.36 | 52.27 | 33.94 | | **+NMP-SVD** | **Coupled calibration + sparse expansion** | **17.31** | **52.00** | **8.73** | **50.53** | Directly applying these existing N:M pruning methods leads to substantial performance degradation, whereas NMP-SVD improves upon the Dense-SVD baseline on both models. These results demonstrate that **increased projection capacity alone is insufficient; realizing the fidelity–capacity trade-off requires optimization that explicitly accounts for the coupled sparse factors and the original weighted reconstruction objective.**
AuthorsOpenReview ↗

Response to Reviewer, Part 2/3

Q2/W2: Additional Evaluation on Challenging Downstream Tasks.

We thank the reviewer for this suggestion. We conducted additional evaluations on mathematical reasoning (MATH-500 and GSM8K), code generation (LiveCodeBench), and instruction following (IFEval), using Qwen3-8B and LLaMA3-8B across three SVD backbones. NMP-SVD uses 2:4 sparsity with ρ=1.5\rho=1.5 (Approximately 75% of the parameter count of dense-SVD).

For calibration, we used 128 reasoning samples covering mathematics, coding, and general instructions, with a maximum length of 8K tokens per sample. Reasoning responses were generated by the target model, and the complete tokenized inputs were retained for activation statistics and compression calibration. Importantly, the calibration data were collected independently of the evaluation benchmarks, with no overlap with MATH-500, GSM8K, LiveCodeBench, or IFEval. All compression methods used the same calibration data to ensure a fair comparison.

NMP-SVD consistently improves all four tasks on Qwen3-8B at 20% compression ratio. For example, with KFAC-SVD, MATH-500 improves from 63.6% to 79.2%, LiveCodeBench from 2.6% to 13.1%, and IFEval from 32.0% to 55.8%. At 40% compression ratio, substantial improvements remain in mathematical reasoning, although code generation is still severely degraded.

These results demonstrate that the benefits of NMP-SVD extend beyond perplexity and standard zero-shot tasks to more challenging generation-intensive capabilities, particularly mathematical reasoning and instruction following.

Table: Performance on challenging downstream tasks for Qwen3-8B (%). IFEval reports prompt-level strict accuracy.

RatioBackboneMethodMATH-500 ↑\uparrowGSM8K ↑\uparrowIFEval ↑\uparrowLiveCodeBench ↑\uparrow
––Original87.294.682.328.7
20%ASVDDense-SVD7.422.222.90.0
+NMP-SVD75.286.472.117.9
SVD-LLMDense-SVD67.286.031.81.9
+NMP-SVD81.692.363.014.9
KFAC-SVDDense-SVD63.686.632.02.6
+NMP-SVD79.291.155.813.1
40%ASVDDense-SVD0.40.38.50.0
+NMP-SVD10.227.927.20.0
SVD-LLMDense-SVD14.233.719.00.0
+NMP-SVD52.873.221.40.0
KFAC-SVDDense-SVD15.437.216.60.0
+NMP-SVD37.866.621.80.0
AuthorsOpenReview ↗

Response to Reviewer, Part 3/3

Q3/W3: Inference Efficiency Under Small Batch-size and Realistic Serving Workloads.

We agree that, when using generic N:M sparse kernels, NMP-SVD does not provide a decoding speedup over Dense-SVD at small batch sizes. This limitation mainly arises from the mismatch between coarse-grained sparse GEMM tiles and the narrow matrix shapes encountered during decoding. Consequently, a large fraction of the computational resources remains underutilized, while kernel execution overhead becomes relatively significant.

To address this, we developed dedicated GPU-Efficient Sparse Decoding Kernels:

(1) Fine-Grained SIMT Kernel (B=1B=1–44). We replace Tensor Core-based matrix multiplication with vector-oriented sparse computation (GEMV-style) using CUDA-core SIMT parallelism. This avoids inefficient GEMM tiling for small batches.

(2) Small-N CUTLASS Kernel (B=8B=8–128128). We retain 2:4 Sparse Tensor Core computation but reduce the tile size from 128×128×64128\times128\times64 to 64×64×6464\times64\times64, increasing GPU parallelism and reducing tile underutilization.

With the optimized kernels, we further evaluated NMP-SVD (ρ=1.4\rho=1.4, compression ratio 40%) across different decoding batch sizes and KV context lengths. The results show that NMP-SVD achieves lower decoding latency and peak memory than Dense-SVD across the evaluated settings.

At B=1B=1 and KV length 2048, decoding latency decreases from 27.94 ms to 24.58 ms (1.137$\times$ speedup), while peak memory decreases from 10.06 GiB to 9.13 GiB. The speedup reaches 1.179$\times$ at B=2B=2 and remains approximately 1.14$\times$ across KV lengths of 256–2048 at B=1B=1.

These results demonstrate that with optimized sparse kernels, NMP-SVD can accelerate decoding even at small batch sizes, extending its practical benefits beyond high-concurrency settings.

In Section 5.5, we have added an introduction to our dedicated GPU-efficient sparse decoding kernels and an experimental analysis of them.

Table: Decoding efficiency with KV context length = 2048 and varying batch size. Time (memory) shown explicitly as ms (GiB).

BatchDenseDense-SVDSparse-SVD (ρ=1.4\rho=1.4)Speedup vs. Dense-SVD ↑\uparrow
124.13 ms  (15.22 GiB)24.13\,\mathrm{ms}\;(15.22\,\mathrm{GiB})27.94 ms  (10.06 GiB)27.94\,\mathrm{ms}\;(10.06\,\mathrm{GiB})24.58 ms  (9.13 GiB)24.58\,\mathrm{ms}\;(9.13\,\mathrm{GiB})1.137×1.137\times
224.12 ms  (15.48 GiB)24.12\,\mathrm{ms}\;(15.48\,\mathrm{GiB})29.03 ms  (10.32 GiB)29.03\,\mathrm{ms}\;(10.32\,\mathrm{GiB})24.63 ms  (9.39 GiB)24.63\,\mathrm{ms}\;(9.39\,\mathrm{GiB})1.179×1.179\times
424.46 ms  (15.99 GiB)24.46\,\mathrm{ms}\;(15.99\,\mathrm{GiB})29.00 ms  (10.83 GiB)29.00\,\mathrm{ms}\;(10.83\,\mathrm{GiB})28.80 ms  (9.90 GiB)28.80\,\mathrm{ms}\;(9.90\,\mathrm{GiB})1.007×1.007\times
826.08 ms  (17.01 GiB)26.08\,\mathrm{ms}\;(17.01\,\mathrm{GiB})29.32 ms  (11.85 GiB)29.32\,\mathrm{ms}\;(11.85\,\mathrm{GiB})27.72 ms  (10.26 GiB)27.72\,\mathrm{ms}\;(10.26\,\mathrm{GiB})1.058×1.058\times
1629.00 ms  (19.06 GiB)29.00\,\mathrm{ms}\;(19.06\,\mathrm{GiB})28.98 ms  (13.90 GiB)28.98\,\mathrm{ms}\;(13.90\,\mathrm{GiB})27.69 ms  (12.31 GiB)27.69\,\mathrm{ms}\;(12.31\,\mathrm{GiB})1.047×1.047\times
3244.87 ms  (23.15 GiB)44.87\,\mathrm{ms}\;(23.15\,\mathrm{GiB})38.81 ms  (17.99 GiB)38.81\,\mathrm{ms}\;(17.99\,\mathrm{GiB})38.32 ms  (16.40 GiB)38.32\,\mathrm{ms}\;(16.40\,\mathrm{GiB})1.013×1.013\times

Table: Decoding efficiency with batch size = 1 and varying KV context length. Time (memory) shown explicitly as ms (GiB).

KV lengthDenseDense-SVDSparse-SVD (ρ=1.4\rho=1.4)Speedup vs. Dense-SVD ↑\uparrow
25624.24 ms  (15.00 GiB)24.24\,\mathrm{ms}\;(15.00\,\mathrm{GiB})28.00 ms  (9.84 GiB)28.00\,\mathrm{ms}\;(9.84\,\mathrm{GiB})24.66 ms  (8.91 GiB)24.66\,\mathrm{ms}\;(8.91\,\mathrm{GiB})1.136×1.136\times
51224.13 ms  (15.03 GiB)24.13\,\mathrm{ms}\;(15.03\,\mathrm{GiB})27.95 ms  (9.87 GiB)27.95\,\mathrm{ms}\;(9.87\,\mathrm{GiB})24.63 ms  (8.94 GiB)24.63\,\mathrm{ms}\;(8.94\,\mathrm{GiB})1.135×1.135\times
102424.18 ms  (15.10 GiB)24.18\,\mathrm{ms}\;(15.10\,\mathrm{GiB})27.98 ms  (9.94 GiB)27.98\,\mathrm{ms}\;(9.94\,\mathrm{GiB})24.59 ms  (9.01 GiB)24.59\,\mathrm{ms}\;(9.01\,\mathrm{GiB})1.138×1.138\times
204824.13 ms  (15.22 GiB)24.13\,\mathrm{ms}\;(15.22\,\mathrm{GiB})27.94 ms  (10.06 GiB)27.94\,\mathrm{ms}\;(10.06\,\mathrm{GiB})24.58 ms  (9.13 GiB)24.58\,\mathrm{ms}\;(9.13\,\mathrm{GiB})1.137×1.137\times

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank you for your careful reviews and constructive feedback. We have carefully addressed the concerns and conducted several additional experiments to strengthen our submission.

Specifically, we have:

  1. Clarified the methodological novelty of NMP-SVD and added comparisons with direct N:M pruning methods applied to both low-rank factors and original weight matrices.
  2. Extended downstream evaluations to mathematical reasoning (MATH-500, GSM8K), code generation (LiveCodeBench), and instruction following (IFEval).
  3. Developed dedicated sparse decoding kernels and demonstrated decoding speedups over Dense-SVD even at small batch sizes, including batch size one.
  4. Supplemented more efficiency and ablation analyses, including compression time and peak memory for 32B/70B models, adaptive dense–sparse selection, and alternating dual-factor calibration.

We have also revised Section 5.5 to incorporate the new kernel designs and decoding experiments, and clarified the experimental settings and methodological limitations in the manuscript. Detailed results and explanations are provided in our individual responses.

We hope these additional experiments and revisions adequately address your concerns. We would greatly appreciate your feedback on whether our responses have resolved the key issues, and we welcome any further questions or suggestions.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.