- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes NMP-SVD, a plug-and-play enhancement for SVD-based LLM compression that uses N:M-structured sparse low-rank factors to retain more projection directions under a limited budget. It combines dual-factor calibration, residual-based projection expansion, and block-wise backfitting, while retaining dense factors for sparsity-sensitive layers. Experiments across multiple models and datasets show improved quality–compression trade-offs, higher prefill efficiency, and lower memory usage.
Critical strengths
- The method combines low-rank decomposition with N:M structured sparsity to retain more projection directions under a limited storage budget, while reducing reconstruction error through alternating dual-factor calibration and residual-guided expansion.
- The experiments cover multiple model families, scales, compression ratios, and SVD backbones, providing broad evidence of the method’s effectiveness and compatibility. Component ablations and end-to-end latency and memory measurements further support its design choices and practical benefits.
Critical weaknesses
- The decoding speedup is limited to high-concurrency settings in the reported experiments, while slower decoding at smaller batch sizes and the absence of batch-one evaluation leave its benefits for interactive deployment unclear.
- The layer-wise selector minimizes local reconstruction error without explicitly accounting for interactions across compressed layers. Consequently, these local choices may not improve the model’s final output distribution.
Questions for authors
- How much additional compression time and peak memory does NMP-SVD require relative to the underlying SVD compressor, particularly for the 32B and 70B models?
- Can author report decoding latency at smaller batch sizes, particularly batch size one, and across different context lengths to clarify the benefits for interactive deployment?
- How does adaptive selection compare with the all-sparse variant under matched storage budgets, and what fraction of layers retain dense representations?
- Could you compare alternative calibration strategies to better isolate the contribution of the proposed alternating dual-factor calibration?
6 comments
Response to Reviewer, Part 2/4
W1/Q2: Small-Batch Decoding Efficiency.
We thank the reviewer for highlighting this limitation. The original decoding experiments employed general-purpose N:M sparse kernels, whose coarse-grained GEMM tiles are inefficient for the narrow matrix shapes encountered at small batch sizes, resulting in low GPU utilization and relatively high execution overhead.
To address this limitation, we developed two dedicated GPU-efficient sparse decoding kernels:
(1) Fine-Grained SIMT Kernel (–). We replace coarse-grained Tensor Core GEMM with fine-grained, vector-oriented sparse computation (GEMV-style) using CUDA-core SIMT parallelism. This better matches small-batch workloads.
(2) Small-N CUTLASS Kernel (–). We retain 2:4 Sparse Tensor Core computation but reduce the GEMM tile size from to , improving parallelism and reducing tile underutilization.
We further evaluated NMP-SVD with these optimized kernels at a 40% base compression ratio and , varying both batch size and KV context length. The results are summarized below.
Decoding efficiency across batch sizes (KV length = 2048). Each entry reports latency (ms) / peak memory (GiB).
| Batch | Dense | Dense-SVD | NMP-SVD | Speedup vs. Dense-SVD |
|---|---|---|---|---|
| 1 | 24.13 / 15.22 | 27.94 / 10.06 | 24.58 / 9.13 | 1.137$\times$ |
| 2 | 24.12 / 15.48 | 29.03 / 10.32 | 24.63 / 9.39 | 1.179$\times$ |
| 4 | 24.46 / 15.99 | 29.00 / 10.83 | 28.80 / 9.90 | 1.007$\times$ |
| 8 | 26.08 / 17.01 | 29.32 / 11.85 | 27.72 / 10.26 | 1.058$\times$ |
| 16 | 29.00 / 19.06 | 28.98 / 13.90 | 27.69 / 12.31 | 1.047$\times$ |
| 32 | 44.87 / 23.15 | 38.81 / 17.99 | 38.32 / 16.40 | 1.013$\times$ |
Decoding efficiency across KV context lengths (). Each entry reports latency (ms) / peak memory (GiB).
| KV Length | Dense | Dense-SVD | NMP-SVD | Speedup vs. Dense-SVD |
|---|---|---|---|---|
| 256 | 24.24 / 15.00 | 28.00 / 9.84 | 24.66 / 8.91 | 1.136$\times$ |
| 512 | 24.13 / 15.03 | 27.95 / 9.87 | 24.63 / 8.94 | 1.135$\times$ |
| 1024 | 24.18 / 15.10 | 27.98 / 9.94 | 24.59 / 9.01 | 1.138$\times$ |
| 2048 | 24.13 / 15.22 | 27.94 / 10.06 | 24.58 / 9.13 | 1.137$\times$ |
With the optimized kernels, NMP-SVD achieves lower decoding latency and peak memory than Dense-SVD across all evaluated settings. At and KV length 2048, latency decreases from 27.94 to 24.58 ms (1.137$\times$ speedup), while peak memory decreases from 10.06 to 9.13 GiB. The speedup reaches 1.179$\times$ at and remains approximately 1.14$\times$ across KV lengths of 256–2048 at . Notably, at , NMP-SVD also approaches the latency of the original dense model while reducing peak memory from 15.22 to 9.13 GiB.
These results demonstrate that dedicated sparse execution kernels can overcome the small-batch inefficiency of generic sparse GEMM implementations, extending the practical benefits of NMP-SVD to low-concurrency decoding. We have updated Section 5.5 to describe the optimized kernels and analyze the additional decoding results.
Response to Reviewer, Part 3/4
W2: Why Local Adaptive Selection Remains Useful Without Global Guarantees.
We thank the reviewer for this comment. NMP-SVD is designed as a plug-and-play enhancement to existing SVD compressors, preserving their original weighting and rank allocation strategies (e.g., SVD-LLM-v2, KFAC-LEMS, and Zero-Sum SVD). Our goal is to improve how the retained projections are represented, rather than introducing a new global compression strategy.
We agree that lower local reconstruction error does not guarantee better final model outputs, because errors may interact and propagate across compressed layers. However, adaptive selection is intended as a simple safeguard rather than a global optimizer. Since sparsity tolerance varies across layers, we retain the original Dense-SVD representation whenever the expanded sparse realization produces higher local weighted reconstruction error. This prevents the selection of a locally worse representation under the inherited reconstruction objective, although it does not guarantee better downstream performance.
Importantly, in our evaluated configurations, the sparse realization requires less storage than its dense counterpart, including sparsity metadata. Thus, adaptive selection never increases storage beyond the original Dense-SVD baseline, while allowing sparsity-tolerant layers to benefit from more approximate projections. This provides a practical safeguard without changing the underlying SVD compression framework.
Q3: Adaptive Selection vs. All-Sparse and Dense Layer Ratios
We thank the reviewer for this suggestion. NMP-SVD does not rely on adaptive selection for its effectiveness, as demonstrated by the all-sparse results in Table 1. We further compare all-sparse and adaptive selection under the same base ranks and expansion ratios in Table 4.
Comparison of all-sparse and adaptive selection (KFAC-SVD, ρ=1.4).
| Model | Reduction | Variant | WT2 ↓ | AVG ↑ | Dense Layers | Dense Ratio |
|---|---|---|---|---|---|---|
| LLaMA2-13B | 20% | All-sparse | 6.38 | 58.71 | 0/280 | 0% |
| Adaptive | 6.28 | 59.26 | 115/280 | 41.07% | ||
| LLaMA2-13B | 40% | All-sparse | 10.06 | 48.26 | 0/280 | 0% |
| Adaptive | 9.88 | 49.30 | 99/280 | 35.36% | ||
| LLaMA3-8B | 20% | All-sparse | 12.49 | 53.71 | 0/224 | 0% |
| Adaptive | 12.18 | 54.20 | 118/224 | 52.68% | ||
| LLaMA3-8B | 40% | All-sparse | 50.67 | 37.10 | 0/224 | 0% |
| Adaptive | 51.21 | 36.54 | 86/224 | 38.39% |
Adaptive selection improves both perplexity and average accuracy in three of the four configurations, although improvements in local reconstruction error do not always translate into better downstream performance.
The fraction of retained dense layers also varies across models and compression levels. For LLaMA2-13B, the dense ratio decreases from 41.07% to 35.36% as base reduction increases from 20% to 40%, while LLaMA3-8B decreases from 52.68% to 38.39%. Appendix C further provides detailed layer-wise selection patterns across different SVD backbones and expansion ratios, demonstrating heterogeneous sparsity tolerance.
We note that these comparisons use identical base ranks and expansion ratios, rather than strictly matched final storage budgets. Nevertheless, adaptive selection remains below the original Dense-SVD storage budget in the evaluated configurations.
Response to Reviewer, Part 1/4
We thank the reviewer for raising this question. NMP-SVD introduces additional offline computation for alternating factor calibration, residual-guided expansion, and backfitting. The total compression times and peak memory are already reported in Appendix D (Table 8). We summarize the results for Qwen3-32B and LLaMA2-70B below.
Total compression time under uniform rank allocation at a 40% base parameter-reduction ratio. NMP-SVD uses 2:4 sparsity and ρ=1.5. Times are in hours/minutes; peak memory (GiB) follows the slash.
For Qwen3-32B and LLaMA2-70B, the peak GPU memory remains nearly unchanged when integrated with SVD-LLM and KFAC-SVD, while ASVD exhibits a larger increase. Nevertheless, the peak memory remains manageable under layer-wise compression. Importantly, these additional optimization costs are incurred only once during offline compression and do not introduce additional computation at inference time.