- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies N:M semi-structured sparsification of LLMs and presents BLADE, a seemingly novel approach to learn both the pruning mask and the reconstruction of non-pruned weights. The paper seems to be in the tradition of MaskLLM. Most interestingly, at least to me, seems to be the insights w.r.t. the use of a magnitude-independent optimizer. The paper provides a computational study on some smaller-scale models to provide empirical proof of their hypotheses.
Critical strengths
Overall, I'm generally favorable towards this paper, since I think that MaskLLM is not the end-of-it-all method, and I greatly appreciate people working on making sparsity at least somewhat attractive again for LLMs. The method is simple, the paper is generally relatively well written and easy to follow.
Critical weaknesses
I have several concerns w.r.t. this paper that make me lean towards rejection. I will outline my concerns below. I will put questions separately. For the moment, I recommend a score of 2, but I'm looking forward to seeing the authors responses. My current interpretation is that this is an interesting paper, with minor-to-medium impact, and a relatively bad execution of the computational study. Let me be more precise:
Contribution
- I might be mistaken since I have not checked MaskLLM in a while, but is your approach not basically MaskLLM without the end-to-end approach. So basically, you are following MaskLLM but work on individual blocks? If that is so, I think the novelty is somewhat limited.
- What I find significantly more interesting than the actual method is the fact that you are using Lion, since it allows you to tune on the first block and then "transfer" to the others. Why are these results so hidden in the appendix and under explored? This should be made much more prominent.
Soundness
- The computational study is badly executed, in my opinion. The models are old and tiny. Every model considered is from 2024 or earlier, pre reasoning. If I am not mistaken, one of the great advantages of your method over MaskLLM is that you have memory savings, so why are you using models <= 8B? I cannot be sure this transfers.
- You compare to Wanda, SparseGPT, ...: are these methods also retrained/finetuned layer wise? This would yield the fair comparison. See [1] for instance.
References
- [1]: Zimmer, Max, et al. "Perp: Rethinking the prune-retrain paradigm in the era of llms." arXiv preprint arXiv:2312.15230 (2023).
Questions for authors
- L145: What's a baked mask? One fresh out of the oven? baked == fixed?
3 comments
I thank the authors for their detailed answer. Most of my concerns have been addressed, but a major concern is still open: I am not convinced that this is not just MaskLLM per block and with a better hyper parameter/optimizer setup. Can you convince me otherwise? Whats the technical novelty other than porting MaskLLM to per-block and using a different optimizer setup?
If this can be convincingly answered, I would raise my score to 3, otherwise I think 2 is the adequate score for this paper.
Dear Reviewer nNS3,
Thank you for the follow-up, and we are glad the other concerns are addressed.
BLADE takes its mask parameterization from MaskLLM and its optimizer from Lion (Chen et al., 2023), and the paper credits both. What it adds is the finding that the direct port of MaskLLM to blocks fails, the reason it fails, and a design built on that reason.
The right setup cannot be found by tuning
If the optimizer were just a hyperparameter, a careful search on per-block MaskLLM would find it. It does not. We gave AdamW (MaskLLM's optimizer), Adam, Muon, SGD, and Lion each its own search on the first block of Qwen2.5-0.5B. All five converge there, and AdamW wins (71.17% error reduction against 70.85% for Lion). Frozen and carried to the first block of LLaMA-3.1-8B, AdamW learns nothing (0.00%, reverted to the prior mask), and over all 24 blocks of Qwen2.5-0.5B its model ends at a C4 KL divergence of 0.604 against 0.377 for Lion (Table-7-Page-31). A tuned port picks the optimizer that fails.
Why it fails
The block-local mask gradient vanishes as the relaxation hardens, by 4× to 525× on that block, and its scale is set by the model: on LLaMA-3.1-8B it is two orders of magnitude smaller than on Qwen2.5-0.5B (Section-2.3-Page-5, Table-8-Page-32). An optimizer whose step follows the gradient's magnitude ties its learning rate to the scale it was tuned at, so its configuration loses its gain on another model and degrades with depth on the same one. A sign step moves every logit by the same amount at any scale. That is why one configuration, tuned once on a 0.5B model, runs unchanged on every block of every model up to the 20B mixture-of-experts GPT-OSS.
What else the block-local setting needs
- Two stages, not one. Training the mask and the weights together, with its own 168-trial search, wins the block it was tuned on and none of the other 23, ending 23% worse in held-out KL divergence than BLADE-Full (
Table-16-Page-37). - Two activation streams. Each block trains on the compressed model's own activations against the dense model's targets. Removing the dense targets from the weight stage drops LLaMA-3.1-8B at 2:4 from 52.7 to 47.6 average accuracy (
Table-34-Page-61). - 4:8 at scale. With 70 candidates per group instead of 6, the signal per candidate is diluted further, and the sign step keeps it usable. BLADE learns 4:8 masks for models up to 20B parameters on one GPU, where MaskLLM runs out of memory beyond 640 GB.
In short, we identify vanishing, model-dependent mask-logit gradients as the obstacle to block-wise mask learning and solve it with a sign-based optimizer; a hyperparameter search over the direct port does not find this solution.
We thank the reviewer for the careful reading, the encouraging view of sparsity research, and the specific suggestions. Changes are marked in blue in the revised paper, and the general response lists them. We address each point below.
Contribution: Is BLADE MaskLLM applied to individual blocks?
What we found is that applying MaskLLM to individual blocks does not work on its own. We identify vanishing mask-logit gradients as the obstacle to a block-wise mask-selection recipe that transfers across blocks and models, and solve it with a sign-based optimizer, Lion. Block-local mask learning runs on a gradient signal that vanishes as the relaxation hardens (on the first block of Qwen2.5-0.5B, the mask-logit gradient norm falls by 4× to 525×,
Section-2.3-Page-5), and AdamW, MaskLLM's optimizer, converges where it is tuned but, carried to the first block of LLaMA-3.1-8B, reverts to the one-shot prior mask. The mask is learned inside an objective where each block takes its inputs from the already compressed model and its targets from the dense model. The response to Reviewer n7j6 gives the full comparison with MaskLLM and with Shin et al. (2024).Contribution: The Lion transfer deserves more prominence
We agree, and the revision moves it into the main text.
Section-2.3-Page-5now states the vanishing-gradient diagnosis with numbers and its consequence: an optimizer that normalizes by gradient magnitude stays tied to the gradient scale it was tuned at.Section-3.4-Page-9reports the transfer: one Lion configuration, tuned once on the first block of Qwen2.5-0.5B, runs unchanged on every block of every model, including the mixture-of-experts GPT-OSS-20B, where, on its first block, it cuts the held-out block error by 86.56% against 74.43% for AdamW (Table-7-Page-31). The introduction now states it as well (Section-1-Page-1).Soundness: The models are old, small, and not reasoning models
We chose LLaMA-2 and LLaMA-3.1 because MaskLLM and the one-shot methods we compare against report on them, but we agree that this was the main gap of the submission. The revision adds three recent reasoning models up to 20B parameters, including a mixture-of-experts model: Qwen3-0.6B and Qwen3-4B (April 2025) and GPT-OSS-20B (August 2025, 21B total and 3.6B active parameters), evaluated in thinking mode on nine generative tasks (
Section-3.2-Page-7,Table-2-Page-9). Mean retention of dense thinking-mode accuracy:The reviewer is right that memory is BLADE's advantage at scale: each of these models compresses on a single B200, in about 6, 26, and 46 to 50 hours (
Section-3.3-Page-9), because only one block is on the GPU at a time. MaskLLM needs eight A100s at 2:4 for a 7B model and runs out of memory beyond 640 GB at 4:8.Soundness: Are the one-shot baselines retrained layer-wise, as in PERP?
They were not retrained in the submission, and we have now added a retraining phase: our weight stage, the same block-local retraining BLADE uses, run on one-shot masks with identical data and settings. On the non-reasoning models at 2:4 (eight-task average accuracy, retention of dense accuracy in parentheses, ↑):
On Qwen3-0.6B at 2:4, we focus on the two stronger one-shot methods, SparseGPT and ADMM (mean retention of dense thinking-mode accuracy over nine generative tasks, ↑):
Retraining closes most of the gap to BLADE-Full, which stays ahead on every model: on Qwen3-0.6B, learning the mask adds 3 to 4 points over retraining alone. The tables are in
Section-3.1-Page-7,Table-33-Page-61, andTable-41-Page-67. PERP (Zimmer et al., 2023) is now discussed inAppendix-A.3-Page-21.L145: What is a "baked" mask?
Yes, "baked" meant fixed: the hard mask after stage one, frozen during weight re-optimization. We agree it was a confusing term, and the revision now uses "fixed" throughout (the only things fresh out of the oven are the rebuttal results).
We thank the reviewer again for the thoughtful feedback, and we hope that our effort in thoroughly addressing all concerns will be meaningfully reflected in the final assessment. We are happy to provide further details during the discussion period.