acceptodds
Under review as a conference paper at ICLR 2027

BLADE: Block-Level Adaptive Mask Discovery with Error Compensation

Abstract

N:M sparsity is the pattern hardware accelerates, but selecting its masks well is either cheap or good, not both: one-shot pruners leave a large quality gap to end-to-end learned masks, while the learned reference, MaskLLM, needs thousands of GPU-hours and hundreds of gigabytes of GPU memory at 2:4, and runs out of memory beyond it. We present BLADE (Block-Level Adaptive mask Discovery with Error compensation), which reaches learned-mask quality at a small fraction of end-to-end cost by recasting whole-model N:M compression as a sequence of block-local solves: each block takes its inputs from the already-compressed stream and trains against the dense model's outputs, so upstream error is absorbed rather than compounded, and each solve runs a differentiable Gumbel-softmax search over the block's per-group keep-patterns followed by re-optimization of the surviving weights under the frozen mask. Block-local mask learning runs on a gradient signal that vanishes as the relaxation hardens; we show that with a fair per-optimizer search, every optimizer still converges where it is tuned, but only the sign-based Lion transfers, carrying one configuration, tuned once on a single block of the smallest model, unchanged across blocks and models. Under a unified evaluation harness spanning one-shot, lightweight-learned, and end-to-end-learned baselines on three models, BLADE matches MaskLLM's 2:4 downstream accuracy with roughly 20x less memory and 56x less compute on a single workstation GPU, and at 4:8, where no end-to-end learned masks exist, it attains the best accuracy, perplexity, and KL divergence in every comparison, 2.4% to 4.0% points above the strongest baseline, with its 4:8 models surpassing even MaskLLM's 2:4 accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
N:M sparsity, semi-structured pruning, model compression, large language models, learnable masks, Gumbel-softmax

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nNS3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies N:M semi-structured sparsification of LLMs and presents BLADE, a seemingly novel approach to learn both the pruning mask and the reconstruction of non-pruned weights. The paper seems to be in the tradition of MaskLLM. Most interestingly, at least to me, seems to be the insights w.r.t. the use of a magnitude-independent optimizer. The paper provides a computational study on some smaller-scale models to provide empirical proof of their hypotheses.

Critical strengths

Overall, I'm generally favorable towards this paper, since I think that MaskLLM is not the end-of-it-all method, and I greatly appreciate people working on making sparsity at least somewhat attractive again for LLMs. The method is simple, the paper is generally relatively well written and easy to follow.

Critical weaknesses

I have several concerns w.r.t. this paper that make me lean towards rejection. I will outline my concerns below. I will put questions separately. For the moment, I recommend a score of 2, but I'm looking forward to seeing the authors responses. My current interpretation is that this is an interesting paper, with minor-to-medium impact, and a relatively bad execution of the computational study. Let me be more precise:

Contribution

  • I might be mistaken since I have not checked MaskLLM in a while, but is your approach not basically MaskLLM without the end-to-end approach. So basically, you are following MaskLLM but work on individual blocks? If that is so, I think the novelty is somewhat limited.
  • What I find significantly more interesting than the actual method is the fact that you are using Lion, since it allows you to tune on the first block and then "transfer" to the others. Why are these results so hidden in the appendix and under explored? This should be made much more prominent.

Soundness

  • The computational study is badly executed, in my opinion. The models are old and tiny. Every model considered is from 2024 or earlier, pre reasoning. If I am not mistaken, one of the great advantages of your method over MaskLLM is that you have memory savings, so why are you using models <= 8B? I cannot be sure this transfers.
  • You compare to Wanda, SparseGPT, ...: are these methods also retrained/finetuned layer wise? This would yield the fair comparison. See [1] for instance.

References

  • [1]: Zimmer, Max, et al. "Perp: Rethinking the prune-retrain paradigm in the era of llms." arXiv preprint arXiv:2312.15230 (2023).
Questions for authors
  • L145: What's a baked mask? One fresh out of the oven? baked == fixed?
3 comments

We thank the reviewer for the careful reading, the encouraging view of sparsity research, and the specific suggestions. Changes are marked in blue in the revised paper, and the general response lists them. We address each point below.


Contribution: Is BLADE MaskLLM applied to individual blocks?

What we found is that applying MaskLLM to individual blocks does not work on its own. We identify vanishing mask-logit gradients as the obstacle to a block-wise mask-selection recipe that transfers across blocks and models, and solve it with a sign-based optimizer, Lion. Block-local mask learning runs on a gradient signal that vanishes as the relaxation hardens (on the first block of Qwen2.5-0.5B, the mask-logit gradient norm falls by 4× to 525×, Section-2.3-Page-5), and AdamW, MaskLLM's optimizer, converges where it is tuned but, carried to the first block of LLaMA-3.1-8B, reverts to the one-shot prior mask. The mask is learned inside an objective where each block takes its inputs from the already compressed model and its targets from the dense model. The response to Reviewer n7j6 gives the full comparison with MaskLLM and with Shin et al. (2024).


Contribution: The Lion transfer deserves more prominence

We agree, and the revision moves it into the main text. Section-2.3-Page-5 now states the vanishing-gradient diagnosis with numbers and its consequence: an optimizer that normalizes by gradient magnitude stays tied to the gradient scale it was tuned at. Section-3.4-Page-9 reports the transfer: one Lion configuration, tuned once on the first block of Qwen2.5-0.5B, runs unchanged on every block of every model, including the mixture-of-experts GPT-OSS-20B, where, on its first block, it cuts the held-out block error by 86.56% against 74.43% for AdamW (Table-7-Page-31). The introduction now states it as well (Section-1-Page-1).


Soundness: The models are old, small, and not reasoning models

We chose LLaMA-2 and LLaMA-3.1 because MaskLLM and the one-shot methods we compare against report on them, but we agree that this was the main gap of the submission. The revision adds three recent reasoning models up to 20B parameters, including a mixture-of-experts model: Qwen3-0.6B and Qwen3-4B (April 2025) and GPT-OSS-20B (August 2025, 21B total and 3.6B active parameters), evaluated in thinking mode on nine generative tasks (Section-3.2-Page-7, Table-2-Page-9). Mean retention of dense thinking-mode accuracy:

ModelStrongest one-shot, 2:4BLADE, 2:4Strongest one-shot, 4:8BLADE, 4:8
Qwen3-0.6B48%72%59%78%
Qwen3-4B72%81%81%85%
GPT-OSS-20B80%91%88%92%

The reviewer is right that memory is BLADE's advantage at scale: each of these models compresses on a single B200, in about 6, 26, and 46 to 50 hours (Section-3.3-Page-9), because only one block is on the GPU at a time. MaskLLM needs eight A100s at 2:4 for a 7B model and runs out of memory beyond 640 GB at 4:8.


Soundness: Are the one-shot baselines retrained layer-wise, as in PERP?

They were not retrained in the submission, and we have now added a retraining phase: our weight stage, the same block-local retraining BLADE uses, run on one-shot masks with identical data and settings. On the non-reasoning models at 2:4 (eight-task average accuracy, retention of dense accuracy in parentheses, ↑):

MethodQwen2.5-0.5BLLaMA-2-7BLLaMA-3.1-8B
SparseGPT34.1 (72%)45.1 (81%)46.3 (76%)
SparseGPT, retrained38.3 (83%)49.2 (89%)52.7 (86%)
BLADE-Full39.0 (84%)49.6 (90%)52.8 (87%)

On Qwen3-0.6B at 2:4, we focus on the two stronger one-shot methods, SparseGPT and ADMM (mean retention of dense thinking-mode accuracy over nine generative tasks, ↑):

MaskRetrainedRetention
SparseGPTno41%
ADMMno48%
SparseGPTyes68%
ADMMyes69%
BLADE-Full (learned)yes72%

Retraining closes most of the gap to BLADE-Full, which stays ahead on every model: on Qwen3-0.6B, learning the mask adds 3 to 4 points over retraining alone. The tables are in Section-3.1-Page-7, Table-33-Page-61, and Table-41-Page-67. PERP (Zimmer et al., 2023) is now discussed in Appendix-A.3-Page-21.


L145: What is a "baked" mask?

Yes, "baked" meant fixed: the hard mask after stage one, frozen during weight re-optimization. We agree it was a confusing term, and the revision now uses "fixed" throughout (the only things fresh out of the oven are the rebuttal results).


We thank the reviewer again for the thoughtful feedback, and we hope that our effort in thoroughly addressing all concerns will be meaningfully reflected in the final assessment. We are happy to provide further details during the discussion period.

I thank the authors for their detailed answer. Most of my concerns have been addressed, but a major concern is still open: I am not convinced that this is not just MaskLLM per block and with a better hyper parameter/optimizer setup. Can you convince me otherwise? Whats the technical novelty other than porting MaskLLM to per-block and using a different optimizer setup?

If this can be convincingly answered, I would raise my score to 3, otherwise I think 2 is the adequate score for this paper.

Dear Reviewer nNS3,

Thank you for the follow-up, and we are glad the other concerns are addressed.

BLADE takes its mask parameterization from MaskLLM and its optimizer from Lion (Chen et al., 2023), and the paper credits both. What it adds is the finding that the direct port of MaskLLM to blocks fails, the reason it fails, and a design built on that reason.

The right setup cannot be found by tuning

If the optimizer were just a hyperparameter, a careful search on per-block MaskLLM would find it. It does not. We gave AdamW (MaskLLM's optimizer), Adam, Muon, SGD, and Lion each its own search on the first block of Qwen2.5-0.5B. All five converge there, and AdamW wins (71.17% error reduction against 70.85% for Lion). Frozen and carried to the first block of LLaMA-3.1-8B, AdamW learns nothing (0.00%, reverted to the prior mask), and over all 24 blocks of Qwen2.5-0.5B its model ends at a C4 KL divergence of 0.604 against 0.377 for Lion (Table-7-Page-31). A tuned port picks the optimizer that fails.

Why it fails

The block-local mask gradient vanishes as the relaxation hardens, by 4× to 525× on that block, and its scale is set by the model: on LLaMA-3.1-8B it is two orders of magnitude smaller than on Qwen2.5-0.5B (Section-2.3-Page-5, Table-8-Page-32). An optimizer whose step follows the gradient's magnitude ties its learning rate to the scale it was tuned at, so its configuration loses its gain on another model and degrades with depth on the same one. A sign step moves every logit by the same amount at any scale. That is why one configuration, tuned once on a 0.5B model, runs unchanged on every block of every model up to the 20B mixture-of-experts GPT-OSS.

What else the block-local setting needs

  • Two stages, not one. Training the mask and the weights together, with its own 168-trial search, wins the block it was tuned on and none of the other 23, ending 23% worse in held-out KL divergence than BLADE-Full (Table-16-Page-37).
  • Two activation streams. Each block trains on the compressed model's own activations against the dense model's targets. Removing the dense targets from the weight stage drops LLaMA-3.1-8B at 2:4 from 52.7 to 47.6 average accuracy (Table-34-Page-61).
  • 4:8 at scale. With 70 candidates per group instead of 6, the signal per candidate is diluted further, and the sign step keeps it usable. BLADE learns 4:8 masks for models up to 20B parameters on one GPU, where MaskLLM runs out of memory beyond 640 GB.

In short, we identify vanishing, model-dependent mask-logit gradients as the obstacle to block-wise mask learning and solve it with a sign-based optimizer; a hyperparameter search over the direct port does not find this solution.

Reviewer_n7j6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes BLADE, an efficient framework for learning N sparsity masks through sequential block-local optimization. It combines MaskLLM-style differentiable mask discovery with error-compensated block reconstruction, sign-based mask optimization, and subsequent weight re-optimization. Experiments on three LLMs demonstrate competitive downstream performance at substantially reduced compression cost compared with end-to-end mask learning, particularly at 2:4 and 4:8 sparsity.

Critical strengths

S1. Relevant problem and encouraging empirical results. The paper addresses the computational expense of learned N pruning. The reported results suggest that block-local learning can reduce compression costs while retaining competitive accuracy.

S2. Clear presentation and useful ablations. The methodology is clearly described, and the paper investigates optimizer transferability, calibration-set size, and joint versus sequential mask–weight optimization.

Critical weaknesses

W1. Limited methodological novelty. The proposed approach appears largely incremental to existing methods. Section 2.2 closely follows MaskLLM (Fang et al., 2024) in its differentiable mask parameterization and Gumbel-softmax optimization, with the primary modification being the replacement of end-to-end training by block-local reconstruction. However, blockwise reconstruction with sequential error compensation is also well established. In particular, Section 2.1 should reference the global propagation approach of Shin et al. (2024), which employs a similar asymmetric reconstruction objective using compressed-model inputs and dense-model targets. The remaining components, including sign-based mask optimization and subsequent weight refinement, also have clear precedents. Overall, the contribution appears to be primarily an integration of existing techniques rather than a substantial methodological advance.

W2. Insufficient evidence of improvement in the efficiency–quality tradeoff. Even with limited methodological novelty, adapting MaskLLM from model-wide to blockwise optimization could be a meaningful contribution if its practical benefits were convincingly demonstrated. However, the evaluation falls short of even basic comparisons established in prior studies of optimization granularity and efficiency–quality tradeoffs (Shin et al., 2024; Lee et al., 2026). In particular, straightforward baselines such as SparseGPT followed by iterative blockwise weight re-optimization are missing, despite the paper explicitly describing such a configuration (BLADEWeight) in Section 2.4. Without this comparison, it is difficult to establish whether BLADE's additional mask-learning complexity provides a meaningful advantage over simpler alternatives.

References

  • Fang et al. (2024). MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models.
  • Shin et al. (2024). Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization.
  • Lee et al. (2026). The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM.
Questions for authors

How does BLADE compare against Wanda or SparseGPT followed by the same weight re-optimization procedure? This would help isolate the benefit of learned-mask discovery under comparable computational budgets.

3 comments

We thank the reviewer for the careful reading and for pointing out where the comparison with prior work needed to be sharper. Changes are marked in blue in the revised paper. Among them, we added three recent reasoning models up to 20B parameters, including a mixture-of-experts model (Qwen3-0.6B, Qwen3-4B, and GPT-OSS-20B): at 4:8, BLADE retains 78%, 85%, and 92% of their dense thinking-mode accuracy, against 59%, 81%, and 88% for the strongest one-shot method (Section-3.2-Page-7, Table-2-Page-9). The general response lists all changes. We address each point below.


W1: Methodological novelty, and the relation to MaskLLM and Shin et al. (2024)

We agree that BLADE builds on MaskLLM's parameterization and on block-wise reconstruction, and the revision credits both (Section-1-Page-1, Section-2.1-Page-3, Section-2.2-Page-4). What is new is what it takes to make learned masks work block by block, and the result that enables:

  1. MaskLLM's recipe does not survive the move to blocks. We identify vanishing mask-logit gradients as the obstacle to a block-wise mask-selection recipe that transfers, and solve it with a sign-based optimizer, Lion. Block-local mask learning runs on a gradient signal that vanishes as the relaxation hardens: on the first block of Qwen2.5-0.5B the mask-logit gradient norm falls by 4× to 525× under every optimizer we tested (Section-2.3-Page-5, Figure-2-Page-31, Table-8-Page-32). AdamW, MaskLLM's optimizer, converges after careful tuning on one model (Qwen2.5-0.5B), but the tuned configuration fails on another (LLaMA-3.1-8B) and reverts to the one-shot prior mask. The sign-based Lion update is insensitive to that scale, so one configuration, tuned once on one block of a 0.5B model, runs unchanged on every block of every model up to GPT-OSS-20B, where, on its first block, it cuts the held-out block error by 86.56% against 74.43% for AdamW (Table-7-Page-31).
  2. BLADE learns the mask inside the reconstruction objective; Shin et al. (2024) keep it fixed. Thank you for the pointer; we now cite and discuss this work in Section-2.1-Page-3 and Appendix-A.3-Page-21. With global propagation, Shin et al. train each pruned block on the pruned model's inputs against the dense model's outputs, the same pairing BLADE uses, but on a SparseGPT, Wanda, or magnitude mask that stays fixed and at unstructured 50% sparsity. BLADE searches the N:M mask itself inside this objective, which is where the vanishing gradients of point 1 arise. W2 below measures what learning the mask adds.
  3. What this enables. BLADE learns 4:8 masks at scales where end-to-end mask learning runs out of memory beyond 640 GB, matches MaskLLM's 2:4 accuracy with about 20× less GPU memory and 49× fewer GPU-hours on the same GPU type, and compresses reasoning models up to 20B parameters on a single GPU (Section-3.1-Page-7, Section-3.2-Page-7, Section-3.3-Page-9).

We also discuss Lee et al. (2026) in Appendix-A.1-Page-20, as the opposite design point: a global objective over 32,768 sequences.


W2/Q: One-shot masks followed by the same weight re-optimization

We ran our weight stage on one-shot masks with the same calibration data, settings, and steps as BLADE. On the non-reasoning models at 2:4 (eight-task average accuracy, retention of dense accuracy in parentheses, ↑):

MethodQwen2.5-0.5BLLaMA-2-7BLLaMA-3.1-8B
SparseGPT34.1 (72%)45.1 (81%)46.3 (76%)
SparseGPT + weight stage38.3 (83%)49.2 (89%)52.7 (86%)
BLADE-Full39.0 (84%)49.6 (90%)52.8 (87%)

On Qwen3-0.6B at 2:4, we focus on the two stronger one-shot methods, SparseGPT and ADMM (mean retention of dense thinking-mode accuracy over nine generative tasks, ↑):

MaskWeight stageRetention
SparseGPTno41%
ADMMno48%
SparseGPTyes68%
ADMMyes69%
BLADE-Full (learned)yes72%

The weight stage on a one-shot mask is a strong configuration, close to the reconstruction of Shin et al. (2024) with global propagation, applied to N:M masks. Learning the mask still adds on top: BLADE-Full retains 72%, against 68% for the weight stage alone from the same SparseGPT prior and 69% from the ADMM mask. On the non-reasoning models, BLADE-Full is ahead of the weight stage alone on all three models, by 0.1 to 0.7 points. At matched hardware, on LLaMA-2-7B at 2:4, the mask stage alone (BLADE-Mask, 9.3 h, 48.8) already matches MaskLLM (48.7, about 785 h on the same GPU), and BLADE-Full (16.2 h, 49.6) is best. We report all configurations in Section-3.1-Page-7, Table-33-Page-61, and Table-41-Page-67, and their hours in Figure-4-Page-41.


We thank the reviewer again for the detailed questions, and we hope that our effort in thoroughly addressing them will be meaningfully reflected in the final assessment. We are happy to provide further details during the discussion period.

Thank you for the clarification and for conducting additional experiments. However, I feel that your response does not sufficiently alleviate my concern about novelty.

I am aware that BLADE constitutes a different algorithm when compared strictly in mechanical and technical terms, but my concern is that these differences do not amount to a sufficiently novel contribution. As I noted in my initial review, while technical details may differ, extending pruning algorithms across different optimization granularities (layerwise, blockwise, and model-wide) has been broadly studied for various pruning algorithms. Extending a specific algorithm in this direction without sufficiently exploiting the change through novel technical developments (along the lines of SparseGPT’s modifications to L-OBS) therefore seems somewhat straightforward and incremental.

Moreover, the gains over the re-optimization baselines appear considerably more marginal than the original comparisons without these baselines suggested, further compounding my concern about the limited novelty.

Although I recognize that identifying and addressing unnecessary inefficiencies in MaskLLM constitutes a real improvement, I believe that the extent and nature of this improvement need to demonstrate greater novelty to justify a standalone contribution. Addressing this concern would, in my view, likely require a substantial revision of the algorithm, which would be beyond the appropriate scope of the review period and better suited to a resubmission.

Nevertheless, I remain open to hearing your perspective and considering further evidence that could help me reassess this concern.

Dear Reviewer n7j6,

Thank you for the follow-up and for staying open to further evidence.

On the gains over re-optimization

We agree with your reading of Table-33-Page-61: once the weights are re-optimized, a one-shot mask comes close to BLADE-Full in accuracy. The contribution we claim is not that margin. It is making learned N:M masks work block by block, which the direct extension of MaskLLM does not do.

The direct extension fails

We take the SparseGPT analogy seriously, because we think BLADE's contribution has the same shape. SparseGPT's step beyond OBS-style pruning was to identify the one obstacle that kept an exact method from scaling and to remove it. Moving MaskLLM's mask search to blocks hits an obstacle of its own. As the relaxation hardens, the mask-logit gradient shrinks by 4× to 525× on the first block of Qwen2.5-0.5B under every optimizer we tested (Section-2.3-Page-5, Table-8-Page-32). MaskLLM's optimizer, AdamW with standard settings and its learning rate tuned on the first block of Qwen2.5-0.5B, then gives no improvement at all on the first block of LLaMA-3.1-8B (0.00%, reverted to the prior mask), and over all blocks of Qwen2.5-0.5B it reaches a C4 KL divergence of 0.604 against 0.377 for Lion (Table-7-Page-31). A sign-based update does not depend on the gradient's scale, so one configuration, tuned once on that block, runs unchanged on every block of every model up to GPT-OSS-20B. We identify vanishing mask-logit gradients as the obstacle to block-wise mask learning and solve it with a sign-based optimizer.

Where the learned mask wins

With the weights left as they are, which is the problem MaskLLM solves, the mask stage alone (BLADE-Mask) is ahead of every one-shot method, including SparseGPT and ADMM, which also update the weights in closed form. This holds in all six non-reasoning settings and in five of six reasoning settings, with a tie in the sixth (Table-1-Page-8, Table-2-Page-9):

ModelPatternMetricBest one-shot (ADMM)BLADE-Mask
Qwen2.5-0.5B2:4Avg. acc.35.337.5
LLaMA-2-7B2:4Avg. acc.46.048.8
LLaMA-3.1-8B2:4Avg. acc.46.651.7
Qwen3-0.6B2:4Thinking retention48%56%
GPT-OSS-20B2:4Thinking retention80%87%
GPT-OSS-20B4:8Thinking retention88%91%

On LLaMA-2-7B, BLADE-Mask (48.8, 9.3 GPU-hours) matches MaskLLM (48.7, about 785 hours on the same GPU type), and across the three models at 2:4 it is within 1.6 points of MaskLLM at block-local cost. At 4:8, where MaskLLM runs out of memory beyond 640 GB, BLADE learns masks for models up to 20B parameters on a single GPU.

We would appreciate it if you would consider these results in your final assessment.

Reviewer_9UySOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BLADE which is a block-level framework for N:M sparsification of pretrained large language models. Instead of learning sparse masks jointly across the entire model, BLADE processes transformer blocks sequentially, minimizing reconstruction error between compressed-block outputs and the corresponding dense-model outputs.

Each block undergoes two stages. First, differentiable mask discovery uses Gumbel-softmax distributions over valid N:M patterns, optimized with the sign-based Lion optimizer to address vanishing mask-logit gradients. Second, the selected mask is fixed and the surviving weights are re-optimized using Muon and AdamW.

Experiments on Qwen2.5-0.5B, LLaMA-2-7B, and LLaMA-3.1-8B compare BLADE against one-shot, lightweight learned-mask, and end-to-end learned-mask methods at 2:4 and 4:8 sparsity. BLADE achieves downstream accuracy comparable to MaskLLM at 2:4 while reporting approximately 56× lower compression GPU-hours and substantially lower GPU memory requirements. At 4:8, BLADE outperforms the evaluated baselines in accuracy, perplexity, and KL divergence. The paper further includes studies of optimizer transferability, calibration-data requirements, mask initialization, block granularity, and compression cost.

Critical strengths
  1. BLADE offers a practical alternative to expensive end-to-end mask learning by performing differentiable mask discovery and weight re-optimization independently. The sequential error-compensation mechanism uses compressed-stream inputs and dense-stream targets, allowing the method to account for errors introduced by preceding blocks. The resulting quality-efficiency tradeoff is compelling: at 2:4 sparsity, BLADE achieves downstream accuracy comparable to MaskLLM while reporting approximately 56× fewer compression GPU-hours and substantially lower GPU memory requirements for the 7B model. This is important because it makes learned-mask sparsification accessible on a single workstation GPU instead of requiring a multi-GPU training setup.

  2. The paper evaluates three pretrained language models across 2:4 and 4:8 sparsity and compares against several one-shot and learned-mask baselines. The experiments include both downstream accuracy and distributional metrics, along with detailed investigations of mask initialization, optimizer transferability, calibration-set size, and block granularity. What is interesting is that the Lion's sign-based updates maintain a transferable optimization configuration despite vanishing mask-logit gradients, whereas several alternatives require more careful tuning. The results at 4:8 sparsity are also notable, showing consistent quality gains over the evaluated baselines and providing evidence that expanding the mask search space can improve compressed-model accuracy.

Critical weaknesses
  1. One of the main empirical findings is that learning 4:8 sparse masks produces noticeably better model quality than 2:4, even though both retain 50% of eligible weights. However, the practical efficiency of these representations depends on hardware and kernel support. While 2:4 sparsity benefits from established GPU Sparse Tensor Core support, equivalent acceleration for general 4:8 patterns is less standardized. The paper acknowledges this limitation but does not provide measurements of inference latency, throughput, or realized speedups for the resulting models.

Consequently, the reported quality improvements at fixed sparsity do not necessarily demonstrate a superior accuracy–latency tradeoff. A 4:8 implementation with additional packing, indexing, or kernel overhead may be slower than an optimized 2:4 implementation. Under matched inference latency, a denser 2:4 model or an alternative compression configuration might achieve comparable or better accuracy.

This is consequential because the primary motivation for N:M sparsity is hardware-efficient deployment. Without an iso-latency analysis or representative deployment benchmarks, the paper convincingly demonstrates improved compression quality but provides limited evidence that its preferred sparsity configurations translate into better practical inference efficiency.

  1. BLADE combines differentiable mask discovery with a second stage that updates all surviving weights, biases, and normalization parameters. While the authors report BLADEMask and BLADEFull separately, the main comparisons against one-shot methods such as SparseGPT, Wanda, and ADMM primarily evaluate these methods without an equivalent block-wise weight re-optimization stage. This makes it difficult to determine how much of the improvement over inexpensive pruning methods is attributable to better mask selection, rather than additional gradient-based optimization.
Questions for authors

Q1. The paper demonstrates substantial accuracy improvements from 4:8 over 2:4 sparsity at the same nominal parameter density. However, the realized inference acceleration of these patterns depends on hardware and kernel support. Could you please provide representative inference latency or throughput measurements for the compressed models, ideally comparing 2:4 and 4:8 at matched inference latency?

Q2 BLADEFull combines learned mask discovery with block-wise weight re-optimization, whereas several one-shot baselines are evaluated without an equivalent refinement stage. Could the authors report results obtained by applying the same second-stage weight optimization to masks generated by SparseGPT or ADMM, using matched calibration data and optimization settings?

Q3 Although BLADE is considerably cheaper than MaskLLM, compression of a 7B–8B model still requires approximately one day on a workstation GPU. Is there an accuracy-versus-compression-compute comparison or quality curves as optimization time increases?

2 comments

We thank the reviewer for the careful reading, the positive assessment, and the constructive suggestions. Changes are marked in blue in the revised paper. Among them, we added three recent reasoning models up to 20B parameters, including a mixture-of-experts model (Qwen3-0.6B, Qwen3-4B, and GPT-OSS-20B): at 4:8, BLADE retains 78%, 85%, and 92% of their dense thinking-mode accuracy, against 59%, 81%, and 88% for the strongest one-shot method (Section-3.2-Page-7, Table-2-Page-9). The general response lists all changes. We address each point below.


W1/Q1: Inference speed of 4:8 against 2:4

The reviewer is correct that higher accuracy at 4:8 does not by itself show a better accuracy-latency trade-off, and the revision no longer reads as if it did. We address the point from three angles:

  1. Comparisons within a pattern are at matched latency. Every 2:4 checkpoint in Table-1-Page-8, ours, MaskLLM's, and the one-shot baselines', uses the same storage format and kernel, so the accuracy differences within a pattern are differences at identical latency and memory. At 2:4, the pattern with hardware support today, BLADE matches MaskLLM and inherits the published 2:4 speedups (Appendix-H-Page-42).
  2. A 4:8 checkpoint is never larger than a 2:4 one. A group of 8 weights has 70 possible patterns (8 choose 4), so 7 index bits per group suffice: (4 × 16 + 7) / (8 × 16) = 0.555 of dense weight memory with 16-bit values, a 1.80× reduction, against 0.5625 (1.78×) for NVIDIA's 2:4 format (Table-20-Page-44). Memory-bound decoding therefore moves no more bytes at 4:8 than at 2:4.
  3. Hardware. No mainstream NVIDIA or AMD GPU accelerates 4:8 sparsity today, so we cannot run an iso-latency comparison of 4:8 against 2:4, and we state this in Section-4-Page-10 and Appendix-H-Page-42. Reporting 4:8 is a standard practice in the field (SparseGPT, Wanda, ALPS, and PERP all report it), because it measures how much quality a richer mask search recovers at the same density. 4:8 sparsity can be implemented in hardware, and strong results are what motivates support for richer patterns in future hardware.

W2/Q2: One-shot masks with the same weight re-optimization

As requested, we ran our weight stage on one-shot masks with the same calibration data, optimizer settings, and step budget as BLADE. On the non-reasoning models at 2:4 (eight-task average accuracy, retention of dense accuracy in parentheses, ↑):

MethodQwen2.5-0.5BLLaMA-2-7BLLaMA-3.1-8B
SparseGPT34.1 (72%)45.1 (81%)46.3 (76%)
SparseGPT + weight stage38.3 (83%)49.2 (89%)52.7 (86%)
BLADE-Full39.0 (84%)49.6 (90%)52.8 (87%)

On Qwen3-0.6B at 2:4, we focus on the two stronger one-shot methods, SparseGPT and ADMM (mean retention of dense thinking-mode accuracy over nine generative tasks, ↑):

MaskWeight stageRetention
SparseGPTno41%
ADMMno48%
SparseGPTyes68%
ADMMyes69%
BLADE-Full (learned)yes72%

The weight stage lifts the SparseGPT and ADMM masks by 21 to 27 points. Learning the mask still adds on top: BLADE-Full retains 72%, against 68% for the weight stage alone from the same SparseGPT prior and 69% from the ADMM mask. On the non-reasoning models, BLADE-Full is ahead of the weight stage alone on all three models, by 0.1 to 0.7 points. The rows are in Section-3.1-Page-7, Table-33-Page-61, and Table-41-Page-67.

Q3: Accuracy against compression compute

We added this view to the paper: one plot per model of accuracy and of held-out KL divergence against compression time on one RTX PRO 6000, with every method as a point (Figure-4-Page-41 for 2:4, Figure-5-Page-42 for 4:8). On LLaMA-2-7B at 2:4 (eight-task average accuracy):

MethodCompression timeAvg Acc (% ↑)
HyperPrune0.4 h44.8
Wanda1.1 h43.5
ADMM1.6 h46.0
SparseGPT2.2 h45.1
MaskPro‡8.9 h45.5
BLADE-Mask9.3 h48.8
BLADE-Full16.2 h49.6
MaskLLM≈785 h†48.7

† MaskLLM reports 1,280 A100 GPU-hours. We converted them with the A100 to RTX PRO 6000 time ratio we measured on BLADE's own workload (1.63×, Appendix-G-Page-39). Timed directly on an A100, BLADE-Full takes about 26 GPU-hours, 49× fewer than MaskLLM.

‡ Excluding its SparseGPT initialization and learning-rate search.

The mask stage alone (BLADE-Mask, 9.3 h, 48.8), at a little over half of BLADE-Full's time, already matches MaskLLM's accuracy, and BLADE-Full improves on it. For the reviewer's question about quality as optimization time grows, we also ran BLADE-Full on Qwen2.5-0.5B at 1/8, 1/4, and 1/2 of its step budget: held-out KL divergence is lower at the full budget than at 1/8 (by 11% at 2:4 and 7% at 4:8), so the extra optimization time buys fidelity to the dense model. A smaller budget is also a cheaper option: at 1/8 of the steps, compression takes 0.40 h instead of 2.0 h at 2:4, with accuracy at the level of the full budget. BLADE's hours are from our current implementation (Table-19-Page-40), and every method's hours are plotted in Figure-4-Page-41.

We thank the reviewer again for the supportive evaluation, and we hope that our effort in thoroughly addressing all questions will be meaningfully reflected in the final assessment. We are happy to provide further details during the discussion period.

Comments on the paper

2 comments

We thank all reviewers for their time and constructive feedback. Reviewer 9UyS found the quality-efficiency trade-off compelling, Reviewers 9UyS and n7j6 found the ablations useful, Reviewer nNS3 found the method simple and the paper easy to follow, and Reviewers 9UyS and nNS3 singled out the transfer of one Lion configuration across blocks and models as an interesting finding.

We have uploaded a revised paper. New or changed text is in blue, and pure trims and moves stay black. The reasoning-model results (Section 3.2, Table 2, Appendix N) and the 4:8 deployment appendix (Appendix H) are new in their entirety, so not every sentence and table cell in them is colored. The main changes are:

  1. Reasoning models (nNS3). We added three recent reasoning models up to 20B parameters, including a mixture-of-experts model: Qwen3-0.6B, Qwen3-4B (both April 2025), and GPT-OSS-20B (August 2025, 21B total and 3.6B active parameters), each compressed on a single GPU. At 4:8, BLADE retains 78%, 85%, and 92% of dense thinking-mode accuracy, against 59%, 81%, and 88% for the strongest one-shot method, and at 2:4, 72%, 81%, and 91% against 48%, 72%, and 80% (Section-3.2-Page-7, Table-2-Page-9).
  2. One-shot masks with our weight stage (9UyS, n7j6, nNS3). We ran our weight re-optimization on SparseGPT masks for the non-reasoning models (Table-33-Page-61) and on SparseGPT and ADMM masks for Qwen3-0.6B (Table-41-Page-67), summarized in Section-3.1-Page-7.
  3. Activation streams (n7j6). We compare our two-stream objective with a single-stream one, and we now cite and discuss Shin et al. (2024), who reconstruct blocks under a fixed mask (Section-2.1-Page-3, Table-34-Page-61).
  4. Lion transfer in the main text (nNS3). The vanishing-gradient diagnosis is now stated with numbers, and the transfer of one configuration from Qwen2.5-0.5B to GPT-OSS-20B is in the main text (Section-2.3-Page-5, Section-3.4-Page-9, Table-7-Page-31).
  5. 4:8 deployment (9UyS). A new appendix gives the weight memory of the 4:8 and 2:4 formats and the state of hardware support (Appendix-H-Page-42, Table-20-Page-44).
  6. Matched calibration. Every one-shot baseline now calibrates on the same data as BLADE (4,096 sequences on the non-reasoning models). The 4:8 claim now covers accuracy only, and its margin changes from 2.4% to 4.0% points to 2.0% to 3.5% points (Section-3.1-Page-7, Table-1-Page-8).
  7. Compute on matched hardware (9UyS). Timed on an A100, the GPU MaskLLM reports, BLADE needs about 26 GPU-hours for LLaMA-2-7B at 2:4, 49× fewer than MaskLLM (Section-3.3-Page-9, Appendix-G-Page-39). The submission's 56× compared GPU-hours across different GPUs.
  8. Missing citations (n7j6, nNS3). Shin et al. (2024), Lee et al. (2026), and PERP (Zimmer et al., 2023) are discussed in Appendix-A.1-Page-20 and Appendix-A.3-Page-21.
  9. Accuracy and held-out KL divergence against compression time (9UyS). One plot per model with every method as a point, plus BLADE at reduced step budgets (Figure-4-Page-41 for 2:4, Figure-5-Page-42 for 4:8), and the same view for the reasoning models, where BLADE retains the most dense thinking-mode accuracy on every model (Figure-6-Page-43, Figure-7-Page-43); the compression hours in Table-19-Page-40 are now from our current implementation.

To make room for these additions within the page limit, the setup and several method details moved to the appendix (Appendix-B-Page-22, Appendix-C-Page-25). The per-reviewer responses below give the new results for each question.

We hope that our effort in thoroughly addressing all concerns will be meaningfully reflected in the final assessment.

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.