acceptodds
Under review as a conference paper at ICLR 2027

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Abstract

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce **SparseDecoding**, a principled *decoding-aware* pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized *N*:*M* sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48× end-to-end wall-clock decoding speedup on A100 GPUs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CpsEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the high latency of LLM decoding caused by memory-bound inference. The authors argue that common Hessian-based training-free pruning methods calibrate on fixed corpora, while real decoding feeds the model self-generated tokens, leading to a distribution shift that hurts pruned models. To mitigate this, the paper proposes SparseDecoding, which builds calibration matrices from activations collected during dense-model autoregressive decoding, making the pruning objective more aligned with decoding behavior. The paper also converts pruned weights into N:M semi-structured sparsity and designs a custom SpMV kernel to speed up decoding. Experiments on Llama and Qwen models cover both unstructured and semi-structured sparsity settings, showing some improvements over fixed-corpus calibration baselines and moderate decoding speedups.

Critical strengths
  • The paper is clearly written. The flow from motivation to method to experiments is easy to follow, and the figures and tables do a good job of conveying the main ideas.

  • The authors analyze the gap between calibration data and decoding data, and give an upper bound on the reconstruction error. This makes the idea of changing the calibration source well grounded rather than just a guess.

Critical weaknesses
  • The core pruning method builds on OBS + SparseGPT backend.

  • Lacks systematic comparison with recent SOTA pruning methods.

  • One problem I have is more about the pruning field in general than this paper alone. Too many pruning papers only compare against other pruning methods, and the whole area ends up looking fancy but not very useful in practice. Another common habit is pushing the sparsity ratio higher and higher, then claiming the method is better because it survives at 50% or 90% sparsity while others collapse. That comparison often means little, because a model that loses most of its ability at extreme sparsity is not something anyone would actually deploy. What is usually missing is a comparison with smaller dense models from the same family. If a pruned large model still cannot beat a smaller model, then pruning does not really save much, because people would just use the smaller one directly. Here it is especially easy to check, since both Qwen and LLaMA have smaller counterparts. The paper would be much more convincing if it showed that the pruned model is better than a smaller dense model in both quality and real cost, including calibration time and kernel implementation effort.

Questions for authors

I have one main question about all pruning methods: is pruning really more meaningful than just using a smaller model?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CpsE (Part 1/3)

We thank the reviewer for the thoughtful review and for recognizing the clarity of our writing. We are glad that the reviewer finds that our analysis of the gap between calibration data and decoding data, with the reconstruction error bound, makes the idea "well grounded rather than just a guess". Below, we respond to each question in turn. Additional results are provided in the tables below.


W1: The core pruning method builds on OBS + SparseGPT backend.

W1-Ans: We thank the reviewer for this observation and agree that SparseDecoding uses an OBS-style solver (SparseGPT by default) as its backend. Our work addresses the mismatch between calibration activations and decode-time activations, a problem orthogonal to pruning solver design that has received limited attention in general non-thinking decoding. We replace activations collected from teacher-forced text with those collected during autoregressive decoding, while keeping the pruning solver unchanged. This defines the reconstruction objective over decode-time activations. Table A shows consistent gains with Wanda, demonstrating that the benefits extend beyond the SparseGPT backend.

Table A: Results on the WritingBench benchmark under 50% and 2:4 sparsity with Wanda on the Qwen3 model family.

ModelSparsityDenseWanda + C4Wanda + SparseDecoding
Qwen3-14B50%5.974.945.23
Qwen3-14B2:45.972.413.30
Qwen3-32B50%6.485.525.71
Qwen3-32B2:46.483.344.39

We hope these clarifications address your concerns.

AuthorsOpenReview ↗

Response to Reviewer CpsE (Part 2/3)

W2: Lacks systematic comparison with recent SOTA pruning methods.

W2-Ans: We thank the reviewer for raising this question.

We compare calibration methods while keeping the model, pruning backend, sparsity pattern, and calibration token budget fixed. SparseDecoding improves over C4 calibration in all reported settings with both SparseGPT and Wanda. We further compare with ROSE [1] and D²Prune [2], using their official implementations with C4 calibration and the same 1M-token calibration budget across methods. SparseDecoding outperforms both methods under 50% and 2:4 sparsity.

Table B: Results on the WritingBench benchmark under 50% and 2:4 sparsity with ROSE, D²Prune, and SparseDecoding on the Qwen3-14B.

ModelSparsityMethodOverall
Qwen3-14B0%Dense5.97
50%D²Prune4.86
ROSE4.98
SparseDecoding (SparseGPT)5.35
2:4D²Prune2.34
ROSE2.24
SparseDecoding (SparseGPT)4.18

We hope these additional results and explanations address your concern.

References

[1] Su and Wang (2026). ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning. CPAL 2026.

[2] Xiong et al. (2026). D²Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awareness. AAAI 2026.

AuthorsOpenReview ↗

Response to Reviewer CpsE (Part 3/3)

W3 & Q1: Too many pruning papers only compare against other pruning methods, and the whole area ends up looking fancy but not very useful in practice.

W3 & Q1-Ans: We thank the reviewer for raising this point.

  1. We agree that many pruning methods stop at reporting sparsity and accuracy without turning them into real efficiency gains, which leaves the impression that pruning is of limited practical use.

  2. The concern that a smaller model can replace a pruned one is well supported for small models and structured pruning, where the smaller network can simply be trained from scratch [3]. For large models, pruning is a practical way to reduce inference cost, since training a smaller model from scratch is expensive. In practice, however, structured pruning often degrades quality, and unstructured or semi-structured sparsity requires a dedicated kernel that most prior work does not provide, so the sparsity rarely turns into real speedup.

  3. This is exactly the gap our work addresses. SparseDecoding aligns calibration with decoding to preserve quality, and our specialized N:M SpMV kernel turns sparsity into measured end-to-end speedups of 1.34 to 1.48× over dense on A100 GPUs, whereas the existing 2:4 Tensor Core path runs at only 0.85 to 0.87× of dense during decoding.

  4. We present the comparison between our pruned Qwen2.5-72B and dense Qwen2.5-32B with the downstream quality and throughput performance as follows:

Table C: Results on the WritingBench benchmark under dense and 16:32 sparsity with SparseDecoding on Qwen2.5-32B and Qwen2.5-72B.

ModelSparsityOverall
Qwen2.5-32B0% (Dense)5.18
Qwen2.5-72B16:325.26

Table D: End-to-end decoding throughput of Qwen2.5-72B with 16:32 sparsity and dense Qwen2.5-32B.

ModelSparsityTokens/s
Qwen2.5-32B0% (Dense)20.4
Qwen2.5-72B16:3229.7

As shown in Table C and Table D, the pruned larger model achieves higher quality and higher throughput than the smaller dense model.

  1. Calibration cost. For Qwen2.5-72B with 1M calibration tokens, calibration takes 59.1 GPU-hours for generation, 29.3 GPU-hours for Hessian collection, and 5.6 GPU-hours for pruning, about 94 A100 GPU-hours in total. It is a one-time cost with no retraining, compared with millions of GPU-hours for pretraining.

  2. Kernel implementation effort. Our kernel is implemented once and can be reused across many models. It is a single Triton implementation, inspired by nmSPARSE and instantiated for each sparsity pattern, with execution configurations autotuned for each matrix shape. Deploying a new model only requires replacing its linear layers and packing the sparse weights, with no new CUDA code.

In summary, pruned Qwen2.5-72B achieves higher quality and higher decoding throughput than dense Qwen2.5-32B, with a one-time calibration cost of about 94 GPU-hours and a reusable kernel. We hope this also answers Q1.


Thank you again for your suggestions. We hope this explanation addresses your concerns. The detailed revisions can be found in Sections 2.2, 3.2, 4.5, and Appendix D (Tables 11-14) in the revised manuscript.

References

[3] Liu et al. (2019). Rethinking the Value of Network Pruning. ICLR 2019.

Reviewer_iuh4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SparseDecoding presents a pruning approach targeted on reducing per token decode latency with (a) an on-policy like calibration step that uses LLM generated decode activations to estimate the hessian for pruning objective and (b) a bitmask-based SpMV triton kernel that better optimizes matrix vector computations that dominate decode forward passes. Experiments show improved downstream performance when compared to a fixed C4-based calibration and improved speedup compared to dense decoding.

Critical strengths
  • The observation that activation discrepancy between pruned and dense model starts small and increases in first few steps eventually reaching a plateau (Figure 1) is very interesting and motivates decoding-aware sparse inference.
  • Authors also implement triton kernels specialized for SpMV operations with N:M sparsity. These, if released, can be helpful for other researchers studying these sparsity settings.
Critical weaknesses
  • Domain mismatch in Calibration datasets and pruning baselines

    • In the results presented in Table 1 and 2. Dense baseline is calibrated on a fixed C4 text, while SparseDecoding uses evaluation specified calibration datasets (LongWriter for writing and LiveCodingBench for coding evals respectively). The presented improvements are a result of both (a) domain specific calibration data and (b) decode-time calibration. Since (b) is the main contribution, isolating is necessary to show improvements. One of the following experiments can isolate these effects:
      • Use fixed C4 text as prompts to generate sequences and uses those decode time activations for calibration (or)
      • Use existing LongWriter/LiveCodingBench traces as fixed text for calibration
    • Authors discuss RAC and RESP as relevant methods that use generated texts for pruning but do not show any comparisons. The description in related work, "Both methods incorporate decode-time information into pruning, while we focus on how the calibration objective differs from the actual decode-time reconstruction objective." is unclear. Authors are requested to either expand the explanation or provide comparisons.
  • Comparison of Triton Kernel Speedup

    • All the speedups are compared against the dense baseline which is expected. Comparing against the existing SpMM (nmSPARSE) kernel without additional modifications is essential to establish improvements.
    • An ablation of additions: bitmask indexing and fixed step bitmask traversal compared to current SpMM implementation would be helpful in isolating speedup gains.
  • Discussion of prior work

    • While using decode-time activations is relatively nascent in pruning-based work, it is somewhat known in quantization based works ([1] for example). It would be helpful to add a section on those approaches in Related Work.

[1] Self-calibration for Language Model Quantization and Pruning.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to Reviewer iuh4 (Part 1/3)

We thank the reviewer for recognizing the motivation from our Figure 1 analysis and the value of our Triton kernels. Below, we respond to each question in turn. In the revised manuscript, we have marked all changes in blue; in this response, we have also highlighted key points for ease of reading.


W1: Domain mismatch in Calibration datasets and pruning baselines.

W1-Ans: We thank the reviewer for raising this concern.

  1. Following the reviewer's suggestion, we conduct an additional experiment using C4 text as prompts for autoregressive generation. This allows us to examine whether the improvement comes primarily from changing the calibration data source.

    We compare three settings using C4 as the common data source: (1) fixed-text calibration, (2) calibration using both prefill and decode activations, and (3) autoregressive calibration using only decode activations. All three settings use the same model, pruning backend, sparsity pattern, and calibration token budget.

     

    Table A: Effect of prompt and decode versus decode-only calibration using C4 prompts under 2:4 sparsity on WritingBench.

     

    ModelSparsityCalibrationOverall ↑
    Qwen3-14B2:4Conventional C41.85
    C4 (Prefill + Decode)3.13
    C4 (Decode-only)3.30

    As shown in Table A, the overall WritingBench score increases from 1.85 to 3.13 with prefill + decode calibration, and further to 3.30 with decode-only calibration. These results show that the improvement is not solely due to the choice of calibration data source. Even when C4 is used as the common data source, autoregressive calibration performs substantially better than conventional fixed-text calibration, with decode-only calibration achieving the highest score.

  2. Relation to RAC and RESP. RAC and RESP study pruning for reasoning models, while SparseDecoding focuses on general decoding in non-thinking mode. RAC reconstructs layer outputs from both prefill and decode activations. SparseDecoding constructs its calibration matrix using decode activations alone. Table A examines this activation choice, with prefill and decode calibration scoring 3.13 and decode-only calibration scoring 3.30. RESP uses first-order gradient-based saliency on traces generated by the model to identify structured units for pruning. SparseDecoding instead applies layer-wise reconstruction through SparseGPT to prune individual weights.

We hope these clarifications address your concerns.

AuthorsOpenReview ↗

Response to Reviewer iuh4 (Part 2/3)

W2: Comparison of Triton Kernel Speedup.

W2-Ans: We thank the reviewer for requesting a direct comparison with nmSPARSE. We compare kernel latency and end-to-end decoding throughput, and provide ablations to quantify the contributions of our optimizations.

  1. Kernel micro-benchmarks. Table B compares our kernel with nmSPARSE on an NVIDIA A100 GPU. We evaluate six matrix shapes and four N:M sparsity patterns under batch-one SpMV execution. Our kernel achieves lower latency in all 24 configurations, with 2.14-4.27× speedup over the evaluated nmSPARSE implementation.

     

    Table B: Kernel latency comparison with nmSPARSE on an NVIDIA A100 GPU.

     

    Weight shapeDense (μs)Kernel2:44:88:1616:32
    4096 × 409627.0nmSPARSE62.3 (0.43×)52.8 (0.51×)49.5 (0.55×)50.8 (0.53×)
    Ours22.2 (1.22×)24.0 (1.13×)23.1 (1.17×)23.4 (1.15×)
    14336 × 409687.8nmSPARSE178.3 (0.49×)151.8 (0.58×)141.0 (0.62×)138.4 (0.63×)
    Ours50.6 (1.74×)50.5 (1.74×)50.4 (1.74×)51.0 (1.72×)
    4096 × 1433680.6nmSPARSE187.5 (0.43×)156.0 (0.52×)141.1 (0.57×)138.0 (0.58×)
    Ours52.3 (1.54×)53.7 (1.50×)53.6 (1.50×)53.7 (1.50×)
    8192 × 819293.3nmSPARSE206.9 (0.45×)172.6 (0.54×)159.4 (0.59×)155.3 (0.60×)
    Ours56.7 (1.65×)59.7 (1.56×)59.6 (1.57×)59.3 (1.57×)
    28672 × 8192289.3nmSPARSE667.9 (0.43×)561.8 (0.51×)512.5 (0.56×)492.1 (0.59×)
    Ours163.6 (1.77×)170.1 (1.70×)170.3 (1.70×)170.7 (1.69×)
    8192 × 28672284.8nmSPARSE699.6 (0.41×)576.0 (0.49×)521.9 (0.55×)505.4 (0.56×)
    Ours164.0 (1.74×)169.7 (1.68×)169.6 (1.68×)170.3 (1.67×)

    While our kernel achieves 1.13-1.77× speedup over dense execution, the nmSPARSE kernel remains slower than dense across all tested configurations. Our design reduces indexing overhead through bitmask representation and fixed-step traversal.

  2. End-to-end comparison with nmSPARSE. To examine whether the kernel-level improvements translate into practical decoding speedups, we further compare end-to-end throughput on Qwen3-14B and Qwen3-32B.

     

    Table C: End-to-end decoding throughput comparison with nmSPARSE on NVIDIA A100 GPUs.

     

    ModelDense (tokens/s)Kernel2:44:88:1616:32
    Qwen3-14B39.8 (1.00×)nmSPARSE20.8 (0.52×)24.0 (0.60×)25.3 (0.64×)26.0 (0.65×)
    Ours53.2 (1.34×)53.9 (1.35×)53.4 (1.34×)53.7 (1.35×)
    Qwen3-32B19.2 (1.00×)nmSPARSE8.6 (0.45×)10.7 (0.56×)11.8 (0.61×)12.1 (0.63×)
    Ours27.8 (1.45×)27.8 (1.45×)27.8 (1.45×)27.8 (1.45×)

    Our kernel achieves 1.34-1.35× speedup over dense execution on Qwen3-14B and approximately 1.45× on Qwen3-32B. The evaluated nmSPARSE implementation reaches 0.52-0.65× of dense throughput on Qwen3-14B and 0.45-0.63× on Qwen3-32B.

Our implementation achieves higher throughput in all eight configurations. These results indicate that the kernel-level improvements also translate into end-to-end decoding performance.

AuthorsOpenReview ↗

Response to Reviewer iuh4 (Part 3/3)

W2: Comparison of Triton Kernel Speedup.

W2-Ans: 3. Additional ablations. We further examine the contributions of bitmask indexing and fixed-step traversal by removing each optimization separately. Both contribute to lower latency across all 24 configurations. Fixed-step traversal provides 1.28-1.56× speedup over row-dependent traversal, while bitmask indexing provides an additional 1.01-1.30× speedup over packed indices.

 

Table D: Kernel ablation on an NVIDIA A100 GPU.

 

Weight shapeDenseKernel variant2:44:88:1616:32
4096 × 409627.0Ours22.2 (1.22×)24.0 (1.13×)23.1 (1.17×)23.4 (1.15×)
w/o bitmask22.9 (1.18×)24.2 (1.12×)23.4 (1.15×)25.0 (1.08×)
w/o fixed-step29.9 (0.90×)30.8 (0.88×)29.9 (0.90×)30.6 (0.88×)
14336 × 409687.8Ours50.6 (1.74×)50.5 (1.74×)50.4 (1.74×)51.0 (1.72×)
w/o bitmask53.7 (1.64×)56.3 (1.56×)56.8 (1.55×)64.5 (1.36×)
w/o fixed-step74.7 (1.18×)75.3 (1.17×)75.4 (1.16×)74.6 (1.18×)
4096 × 1433680.6Ours52.3 (1.54×)53.7 (1.50×)53.6 (1.50×)53.7 (1.50×)
w/o bitmask54.2 (1.49×)57.4 (1.40×)57.4 (1.40×)63.8 (1.26×)
w/o fixed-step74.5 (1.08×)75.0 (1.07×)74.6 (1.08×)74.5 (1.08×)
8192 × 819293.3Ours56.7 (1.65×)59.7 (1.56×)59.6 (1.57×)59.3 (1.57×)
w/o bitmask57.8 (1.61×)63.6 (1.47×)64.6 (1.44×)71.7 (1.30×)
w/o fixed-step83.3 (1.12×)83.3 (1.12×)83.3 (1.12×)83.9 (1.11×)
28672 × 8192289.3Ours163.6 (1.77×)170.1 (1.70×)170.3 (1.70×)170.7 (1.69×)
w/o bitmask166.0 (1.74×)185.5 (1.56×)184.4 (1.57×)218.2 (1.33×)
w/o fixed-step254.2 (1.14×)255.4 (1.13×)255.4 (1.13×)255.7 (1.13×)
8192 × 28672284.8Ours164.0 (1.74×)169.7 (1.68×)169.6 (1.68×)170.3 (1.67×)
w/o bitmask165.0 (1.73×)187.0 (1.52×)185.0 (1.54×)221.3 (1.29×)
w/o fixed-step255.2 (1.12×)255.8 (1.11×)255.7 (1.11×)256.0 (1.11×)

Overall, these results show that our kernel achieves lower SpMV latency than the evaluated nmSPARSE and provides end-to-end speedups in the small-batch decoding setting targeted by our work.

W3: Discussion of prior work.

W3-Ans: We thank the reviewer for pointing out this relevant line of work. We have expanded Section 2.2 (Calibration Data for Pruning) to discuss previous approaches using model-generated calibration data for quantization and pruning.

In particular, Williams et al. [1] propose self-calibration by generating text from a beginning-of-sequence (BOS) token without external prompts, and evaluate their approach for both quantization and pruning. In contrast, SparseDecoding uses task-conditioned autoregressive generation and collects only decode-step activations for layer-wise weight reconstruction. We have revised the related work to acknowledge this approach and clarify the methodological differences.

References

[1] Williams et al. (2025). Self-calibration for Language Model Quantization and Pruning. NAACL 2025.


Thank you again for your suggestions. We hope this explanation addresses your concerns. The detailed revisions can be found in Sections 2.2, 3.2, 4.5, and Appendix D (Tables 11-14) in the revised manuscript.

Reviewer_efuROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SparseDecoding, a framework combining decoding-aware calibration with an optimized sparse execution kernel for LLM inference. The calibration procedure collects layer inputs during autoregressive generation by the dense model, excludes prefill activations, and is compatible with existing pruning solvers such as SparseGPT. The system component implements an N:M SpMV kernel using bitmask indexing and fixed-step traversal. Experiments across four models evaluate writing and code generation, and report decoding speedups of 1.34–1.48× over dense execution on A100 GPUs.

Critical strengths

The paper explicitly considers the potential mismatch between fixed-text calibration and the activations encountered during autoregressive generation. Its decoding-aware calibration integrates with existing pruning workflows, with quality improvements demonstrated using both SparseGPT and Wanda. Compared with the evaluated fixed-corpus baselines, the approach improves writing and code-generation performance. The accompanying N:M SpMV kernel also provides measured decoding speedups over dense execution on A100 GPUs. Together, these results demonstrate the practical value of considering both generation quality and execution efficiency when pruning LLMs.

Critical weaknesses
  1. Insufficient comparisons and ablations. Prior methods such as RAC and RESP already incorporate decode-time information into pruning. Although the paper discusses these methods, it provides neither direct experimental comparisons nor sufficiently detailed analysis to establish whether and by how much SparseDecoding outperforms them, and what accounts for any performance differences. Meanwhile, the main comparisons change both the calibration data source and the activation-selection strategy. It is therefore unclear how much of the improvement comes specifically from excluding prefill activations. Adding more comparisons with these related works and ablations of each component would help readers better understand the performance gains.

  2. The N:M SpMV kernel's contribution is limited. nmSPARSE already provides a dedicated N:M SpMV kernel, and bitmap-based sparse representations have also been explored. While the implementation may be useful, its novelty appears limited, and the evaluation lacks direct performance comparisons against nmSPARSE. Comparisons with compact explicit-index representations and component ablations would also help establish the benefits of bitmap encoding and distinguish them from those of other implementation optimizations.

Furthermore, the reported performance evaluation is limited to batch-one decoding. Batched decoding is important for LLM serving, and larger batch sizes increase opportunities for weight reuse and shift the linear-layer workload from SpMV toward SpMM. It is therefore unclear whether the reported speedups carry over to batched decoding, leaving the kernel’s applicability to broader LLM serving workloads uncertain.

Minor presentation suggestion: Section 3 and Section 3.2 have the identical title, which makes the hierarchy and the scope of the subsection unclear.

Questions for authors
  1. Comparison with prior decoding-aware pruning and ablation: How does the method compare with RAC and RESP? What is the benefit of excluding prefill activations?

  2. Kernel comparisons and batched decoding: How does the proposed kernel compare directly with nmSPARSE’s SpMV? What’s the performance for batched decoding? What’s the largest batch size that speedups are still observed?

5 comments
AuthorsOpenReview ↗

Response to Reviewer efuR (Part 1/4)

We thank the reviewer for recognizing the practical value of considering both generation quality and execution efficiency, and for acknowledging that our decoding-aware calibration integrates with existing pruning workflows and improves writing and code-generation performance. Below, we respond to each question in turn. In the revised manuscript, we have marked all changes in blue; in this response, we have also highlighted key points for ease of reading.


W1: Insufficient comparisons and ablations with RAC and RESP, and unclear how much of the improvement comes from excluding prefill activations.

W1-Ans: We thank the reviewer for raising this question. We agree that the contribution of each factor should be separated, and we add the following experiments and analysis, marked in blue in the revised PDF.

  1. Separating the data source, decode-time activations, and prefill. We add a controlled comparison on Qwen3-14B under 2:4 sparsity in Table A, reporting the WritingBench overall score. All three settings use C4 as the calibration domain, so only the way activations are collected changes.

Table A: Effect of prompt and decode versus decode-only calibration using C4 prompts under 2:4 sparsity on WritingBench.

ModelSparsityCalibrationOverall
Qwen3-14B2:4C41.85
C4 (Prefill + Decode)3.13
C4 (Decode-only)3.30
  1. Results. Replacing fixed-text with decode-time activations raises the score from 1.85 to 3.30 with the calibration domain unchanged, which isolates the effect of the activations from that of the data domain. Excluding prefill activations gives slightly better quality (3.30 versus 3.13 when prefill is kept).

  2. Relation to RAC and RESP. RAC and RESP focus on pruning reasoning models, whereas SparseDecoding targets general decoding in non-thinking mode. RAC reconstructs layer outputs using both prompt (prefill) and decode activations, whereas SparseDecoding uses only decode activations to construct the calibration matrix. The "C4 (Prefill + Decode)" row in Table A isolates this RAC-style activation choice within our pipeline, under the same model, solver, and token budget. It scores 3.13, compared with 3.30 for decode activations only. RESP performs structured pruning with first-order gradient-based saliency, whereas SparseDecoding prunes individual weights through layer-wise reconstruction with SparseGPT. Because the two methods differ in pruning granularity, RESP is not directly comparable in our N:M setting, so we do not include it in Table A.

We appreciate your suggestion and hope these additional results and explanations address your concern.

AuthorsOpenReview ↗

Response to Reviewer efuR (Part 2/4)

W2: The N:M SpMV kernel's contribution is limited.

W2-Ans: We thank the reviewer for raising this point. Motivated by nmSPARSE, we develop an N:M SpMV kernel specifically optimized for LLM decoding, combining bitmask indexing with fixed-step traversal to reduce indexing and control-flow overhead. To establish the contributions of these optimizations, we added a direct comparison with nmSPARSE and ablations of each component on NVIDIA A100 GPUs.

  1. Component ablation. Across the same 24 configurations, our full kernel has lower latency than both ablated variants in every case, and each component has its own contribution.

    • Bitmask indexing. Compared with a compact explicit index, which stores each nonzero’s position within its group as a small integer, the bitmask provides 1.01 to 1.30× speedup. The gain increases with the group width, reaching up to 1.30× at 16:32.

    • Fixed-step traversal. Compared with a variant that keeps the bitmask but loops until each row’s mask is empty, the fixed-step traversal provides 1.28 to 1.56× speedup.

Table B: Kernel ablation on an NVIDIA A100 GPU. Each cell reports latency in μs (speedup over dense execution in parentheses).

Weight shapeDenseKernel variant2:44:88:1616:32
4096 × 409627.0Ours22.2 (1.22×)24.0 (1.13×)23.1 (1.17×)23.4 (1.15×)
w/o bitmask22.9 (1.18×)24.2 (1.12×)23.4 (1.15×)25.0 (1.08×)
w/o fixed-step29.9 (0.90×)30.8 (0.88×)29.9 (0.90×)30.6 (0.88×)
14336 × 409687.8Ours50.6 (1.74×)50.5 (1.74×)50.4 (1.74×)51.0 (1.72×)
w/o bitmask53.7 (1.64×)56.3 (1.56×)56.8 (1.55×)64.5 (1.36×)
w/o fixed-step74.7 (1.18×)75.3 (1.17×)75.4 (1.16×)74.6 (1.18×)
4096 × 1433680.6Ours52.3 (1.54×)53.7 (1.50×)53.6 (1.50×)53.7 (1.50×)
w/o bitmask54.2 (1.49×)57.4 (1.40×)57.4 (1.40×)63.8 (1.26×)
w/o fixed-step74.5 (1.08×)75.0 (1.07×)74.6 (1.08×)74.5 (1.08×)
8192 × 819293.3Ours56.7 (1.65×)59.7 (1.56×)59.6 (1.57×)59.3 (1.57×)
w/o bitmask57.8 (1.61×)63.6 (1.47×)64.6 (1.44×)71.7 (1.30×)
w/o fixed-step83.3 (1.12×)83.3 (1.12×)83.3 (1.12×)83.9 (1.11×)
28672 × 8192289.3Ours163.6 (1.77×)170.1 (1.70×)170.3 (1.70×)170.7 (1.69×)
w/o bitmask166.0 (1.74×)185.5 (1.56×)184.4 (1.57×)218.2 (1.33×)
w/o fixed-step254.2 (1.14×)255.4 (1.13×)255.4 (1.13×)255.7 (1.13×)
8192 × 28672284.8Ours164.0 (1.74×)169.7 (1.68×)169.6 (1.68×)170.3 (1.67×)
w/o bitmask165.0 (1.73×)187.0 (1.52×)185.0 (1.54×)221.3 (1.29×)
w/o fixed-step255.2 (1.12×)255.8 (1.11×)255.7 (1.11×)256.0 (1.11×)
AuthorsOpenReview ↗

Response to Reviewer efuR (Part 3/4)

W2: The N:M SpMV kernel's contribution is limited.

W2-Ans: 2. Comparison with nmSPARSE.

  • Kernel micro-benchmarks. Table C compares batch-one SpMV latency on an NVIDIA A100 across six matrix shapes and four N:M sparsity patterns. Our kernel is faster in all 24 cases, achieving 2.14-4.27× speedup over the evaluated nmSPARSE implementation.

Table C: Kernel latency comparison with nmSPARSE on an NVIDIA A100 GPU. Entries report batch-one SpMV latency in μs (speedup over dense execution in parentheses). All sparse patterns retain 50% of weights.

Weight shapeDense (μs)Kernel2:44:88:1616:32
4096 × 409627.0nmSPARSE62.3 (0.43×)52.8 (0.51×)49.5 (0.55×)50.8 (0.53×)
Ours22.2 (1.22×)24.0 (1.13×)23.1 (1.17×)23.4 (1.15×)
14336 × 409687.8nmSPARSE178.3 (0.49×)151.8 (0.58×)141.0 (0.62×)138.4 (0.63×)
Ours50.6 (1.74×)50.5 (1.74×)50.4 (1.74×)51.0 (1.72×)
4096 × 1433680.6nmSPARSE187.5 (0.43×)156.0 (0.52×)141.1 (0.57×)138.0 (0.58×)
Ours52.3 (1.54×)53.7 (1.50×)53.6 (1.50×)53.7 (1.50×)
8192 × 819293.3nmSPARSE206.9 (0.45×)172.6 (0.54×)159.4 (0.59×)155.3 (0.60×)
Ours56.7 (1.65×)59.7 (1.56×)59.6 (1.57×)59.3 (1.57×)
28672 × 8192289.3nmSPARSE667.9 (0.43×)561.8 (0.51×)512.5 (0.56×)492.1 (0.59×)
Ours163.6 (1.77×)170.1 (1.70×)170.3 (1.70×)170.7 (1.69×)
8192 × 28672284.8nmSPARSE699.6 (0.41×)576.0 (0.49×)521.9 (0.55×)505.4 (0.56×)
Ours164.0 (1.74×)169.7 (1.68×)169.6 (1.68×)170.3 (1.67×)
  • End-to-end decoding. Table D reports throughput on Qwen3-14B and Qwen3-32B. Our kernel achieves 1.34-1.35× the dense throughput on Qwen3-14B and approximately 1.45× on Qwen3-32B, compared with 0.52-0.65× and 0.45-0.63× for the evaluated nmSPARSE implementation. Our kernel has higher throughput in all eight configurations.

Table D: End-to-end decoding throughput comparison with nmSPARSE on NVIDIA A100 GPUs.

ModelDense (tokens/s)Kernel2:44:88:1616:32
Qwen3-14B39.8 (1.00×)nmSPARSE20.8 (0.52×)24.0 (0.60×)25.3 (0.64×)26.0 (0.65×)
Ours53.2 (1.34×)53.9 (1.35×)53.4 (1.34×)53.7 (1.35×)
Qwen3-32B19.2 (1.00×)nmSPARSE8.6 (0.45×)10.7 (0.56×)11.8 (0.61×)12.1 (0.63×)
Ours27.8 (1.45×)27.8 (1.45×)27.8 (1.45×)27.8 (1.45×)

We hope these clarifications address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.