acceptodds
Under review as a conference paper at ICLR 2027

GroupMask: Layer-Adaptive Group-wise Sparsity for Semi-Structured LLM Pruning

Abstract

Semi-structured pruning compresses large language models (LLMs) while keeping a regular sparse structure, but the prevailing N:M pattern fixes the same local sparsity ratio in every layer. Layer-adaptive sparsity allocation improves unstructured pruning, yet it has been reported to be less effective under N:M sparsity, leaving open whether adaptive allocation is of limited value for semi-structured pruning in general or only under the fine-grained N:M pattern. We examine this question with group-level sparsity, which partitions each weight matrix into regular groups, retains or prunes each group as a whole, and allows each layer's sparsity ratio to vary under a global budget. We propose GroupMask, which generates the group selectors of all layers with a lightweight hypernetwork, relaxes them with a Gumbel-Sigmoid parameterization and a straight-through estimator, and learns them through sparsity-budget regularization and self-distillation while keeping the pretrained weights frozen. On LLaMA-2-7B at 50% sparsity with the same group size, learned layer-adaptive allocation reduces WikiText-2 perplexity from 10.02 to 8.30 and raises the average zero-shot accuracy from 0.455 to 0.496 relative to a uniform per-layer ratio. GroupMask obtains the lowest WikiText-2 perplexity on LLaMA-2-7B and the highest average zero-shot accuracy with Alpaca calibration among the evaluated baselines on five LLaMA and Qwen models. Our code is available at https://anonymous.4open.science/r/GroupMask-D037.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model compression, semi-structured pruning, sparsity allocation, hypernetwork, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hBd8OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GroupMask, a layer-adaptive group-wise sparsification method for pruning pretrained LLMs. It learns group masks and allocates a global sparsity budget across layers and projections, aiming to preserve more important parameters than uniform pruning. The method uses differentiable mask learning while keeping the pretrained weights frozen. Experiments on several LLM families evaluate perplexity, downstream-task accuracy, and different sparsity configurations. The main claim is that adaptive group-wise sparsity improves pruning quality.

Critical strengths
  1. The paper studies an important limitation of uniform pruning. A global sparsity budget does not need to be distributed equally across layers and projections. Learning this allocation is a reasonable way to preserve parameters where they are more useful.

  2. The adaptive-versus-uniform ablation provides useful evidence. The comparison uses the same group structure, calibration data, sparsity target, and training schedule for the tested LLaMA-2 7B setting. This helps isolate the contribution of adaptive allocation.

  3. The evaluation includes multiple model families and ablations. The experiments cover LLaMA and Qwen models, with several sparsity settings and analyses of group shape and calibration data. This provides useful evidence about the method’s behavior beyond a single configuration.

Critical weaknesses
  1. The practical inference benefit is not sufficiently demonstrated. The main experiments use a 1×256 group structure, which is different from the standard 2:4 pattern commonly supported by sparse inference hardware. The reported kernel-level speedup is only around 1.09× at 50% sparsity, and end-to-end model latency and throughput are not sufficiently established. Therefore, the results mainly demonstrate pruning quality, not a clear deployment advantage.

  2. Comparisons with existing methods do not consistently match deployment costs and experimental settings. Equal parameter sparsity does not imply equal FLOPs or inference latency when the sparsity structures differ. Some comparisons also use different calibration recipes. These differences make it difficult to determine whether the reported gains reflect better allocation or differences in the experimental setup.

  3. The comparison with learned-mask methods is incomplete, and the generality of the allocation gains remains uncertain. MaskLLM is omitted from the main comparison because of its higher training cost. Although this cost is relevant, a quality-versus-training-cost comparison would be more informative. In addition, the controlled adaptive-versus-uniform evidence is concentrated on LLaMA-2 7B, and variability across independent mask-learning runs is not sufficiently established.

Questions for authors
  1. Can you report end-to-end inference latency and throughput for exported GroupMask models, compared with standard 2:4 pruning methods at matched hardware, batch size, and model quality or measured latency?

  2. Can you provide comparisons with relevant pruning baselines using the same calibration data and clearly matched sparsity or compute budgets, and include a quality-versus-training-cost comparison with MaskLLM or another learned-mask method?

  3. Does the adaptive-versus-uniform improvement remain consistent across additional model families and independent mask-learning runs? Can you report the variability of the main results across seeds?

6 comments

We thank the reviewer for focusing on deployment, cost matching and robustness. All three points led to new experiments, and we report what we found, including where it does not favor GroupMask. Changes are marked in blue; page numbers refer to the revised PDF.

W1 / Q1. End-to-end inference against 2:4 at matched hardware and batch size.

All runs are BF16 full requests (prefill plus 128 generated tokens) with the learned GroupMask masks: static KV cache, CUDA Graph decoding, median of five runs, same RTX PRO 6000.

Speedup over dense (B · prompt)B1·128B1·2048B8·128B8·2048Peak mem. (B1·128)
Dense1.00×1.00×1.00×1.00×13.86 GB
cuSPARSELt 2:41.08×1.10×1.02×1.02×8.21 GB
Triton 2:4 (tuned)1.14×1.08×0.62×0.75×8.57 GB
Hybrid 2:41.18×1.16×0.63×0.76×–
DISP-LLM (structured)1.33×1.23×1.04×1.01×7.44 GB
GroupMask 1×641.49×1.32×1.31×1.13×7.82 GB

Sources: Figure 4 (p. 9) and Sec. 3.6 (p. 8–9); Table 8 (p. 20); protocol in App. C (p. 16–17).

  • Weight values are random in the main benchmark, since they do not change memory access. Rerunning with all real weights (our exported checkpoint, dense LLaMA-2-7B, the official ProxSparse checkpoint) and a natural-text prompt changes every speedup by at most 0.02×.
  • The compressed model reproduces the masked model exactly: identical perplexity and identical greedy outputs (App. C, p. 17).
  • In these runs prefill is eager. Capturing prefill in a CUDA graph as well removes cuSPARSELt's per-call overhead at prompt 128 (103 → 14 ms). The fastest 2:4 path then reaches 1.25× / 1.18× / 1.06× / 1.03×, versus 1.56× / 1.35× / 1.33× / 1.16× for GroupMask in the same run, so the ranking is unchanged (App. C, "Graph-captured prefill", p. 18).
  • 1×256. We moved the main configuration from 1×256 to 1×64, which is more accurate (8.21 vs. 8.69 PPL; App. F, p. 21) at nearly the same speed. On an A100, 1×256 runs at 1.31–1.44× dense, and 1×64 at 1.36× vs. 1.41× for 1×256 at prompt 128 (App. C, "Group width" paragraph, p. 17).

Matched model quality. At 50% sparsity the 2:4 methods are more accurate, so GroupMask is not faster than 2:4 at equal quality. What it offers is a different point on the tradeoff:

LLaMA-2-7BZero-shot ↑Speedup B1·128Peak mem.
Best 2:4 (ProxSparse)0.607≤ 1.18×8.21 GB (cuSPARSELt)
DISP-LLM (structured)0.475 (timed checkpoint: 0.463)1.33×7.44 GB
GroupMask 1×64, 50%0.5141.49×7.82 GB
GroupMask 1×64, 60% (no 2:4 equivalent)0.4541.69×6.48 GB
GroupMask 1×64, 70% (no 2:4 equivalent)0.4311.87×5.14 GB

Sources: Figure 4(b), Table 8 (p. 20); Table 5 (p. 9) and "Effect of sparsity ratio", Sec. 3.5 (p. 8).

Limits (Sec. 5, p. 10; App. C, p. 17):

  • Prefill is not accelerated.
  • On A100 and B200, GroupMask is slower than dense at batch eight.
  • Speedups are compared at equal sparsity, not equal accuracy.

W2 / Q2. Matched calibration, cost and learned-mask baselines.

Deployment cost. We now compare measured latency and memory, not parameter counts (table above). This includes a structured baseline timed with its own architecture (Table 8, footnote §, p. 20).

Calibration. SparseGPT and Wanda are re-run with the same WikiText-103 data as GroupMask (Sec. 3.1, p. 5). The original submission cited C4-calibrated numbers, which we reproduce under C4 (App. G, last paragraph, p. 22).

LLaMA-2, 50% (WikiText calibration unless noted)PPL 7B ↓PPL 13B ↓Zero-shot 7B ↑
SparseGPT 2:48.136.670.592
Wanda 2:410.537.800.579
ProxSparse 2:4 (learned, official C4 checkpoint)8.516.620.607
GroupMask 1×648.217.260.514
GroupMask 1×64 + LoRA7.897.030.519

Sources: Table 1 (p. 6); Table 8 (p. 20). We removed the submission's claim of the best perplexity. ProxSparse is evaluated from its official checkpoints, calibrated on C4 (400 × 4096 tokens), and we did not re-calibrate it on WikiText. With C4 calibration for both, ProxSparse leads GroupMask on LLaMA-2-7B: 8.51 vs. 9.88 PPL and 0.607 vs. 0.552 zero-shot ("Group granularity and calibration data", Sec. 3.5, p. 8; Table 8, p. 20).

Training cost and calibration volume.

MethodToken presentationsHardware / time
GroupMask81.92M1 B200, 4.2–4.5 h
SparseGPT, Wanda0.52Mone-shot

Sources: Table 13 (p. 22); Table 6 (p. 16).

To check whether this difference drives the comparison, we gave the one-shot methods 16× more calibration data:

Calibration sequences (tokens)SparseGPT PPL / zero-shotWanda PPL / zero-shot
128 (0.5M)8.13 / 0.59210.53 / 0.579
512 (2.1M)8.06 / 0.58410.52 / 0.576
1024 (4.2M)8.09 / 0.59310.50 / 0.578
2048 (8.4M)8.06 / 0.59210.57 / 0.576

Source: App. G, last paragraph (p. 22). They are saturated.

We could not run MaskLLM, or a full quality-versus-training-cost curve, within the rebuttal period, and we do not claim a comparison with it.

W3 / Q3. Generality and run-to-run variability.

Adaptive vs. uniform (independent runs)SeedsPPL ↓Zero-shot ↑
LLaMA-2-7B3 + 38.21 ± 0.04 vs. 9.11 ± 0.020.514 ± 0.008 vs. 0.483 ± 0.003
LLaMA-3-8B (GQA)3 + 312.18 ± 0.08 vs. 13.84 ± 0.090.505 ± 0.005 vs. 0.466 ± 0.002
LLaMA-2-7B, 8×8 groups38.15 ± 0.020.521 ± 0.002

Sources: Table 3 and Sec. 3.4 (p. 6–7); Table 4 caption (p. 8).

  • On LLaMA-2-7B, the perplexity gap is about 20× the seed standard deviation.
  • On LLaMA-3-8B, the gap holds even though the uniform models keep slightly more parameters.
  • With the group pattern fixed, uniform, OWL per-block, learned per-block and learned per-projection allocation reach 9.11, 9.09, 8.86 and 8.21 PPL (three seeds each) (Table 3, p. 7). The gain therefore comes mainly from per-projection budgets.
  • All LLaMA-2-7B variants in these comparisons export within 1.6×10−41.6\times10^{-4} of 50% sparsity (App. B, last paragraph, p. 16).

We hope these additions address the reviewer's concerns about practicality and fairness, and we would be glad to provide further details.

Dear Reviewer hBd8,

We thank you again for the thoughtful and constructive feedback. We have carefully considered the concerns raised in the review and provided detailed responses, clarifications, and additional evidence where applicable. As the discussion period is approaching its end, we would appreciate any further feedback on whether our responses adequately address your main concerns, or if any questions remain. We would be happy to provide further clarification or make necessary revisions during the remaining discussion period.

Kind Regards, The Authors

Reviewer_yPS7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether layer-adaptive sparsity allocation, reported to be ineffective under N:M sparsity, is limited for semi-structured pruning in general or only by N:M’s fixed local ratio. It proposes GroupMask, which prunes regular weight groups (e.g., 1256) under a global budget, so per-layer ratios can vary. A GRU hypernetwork generates the group selectors, which are trained with Gumbel-Sigmoid/STE, a sparsity regularizer, and self-distillation, while the pretrained weights stay frozen. On LLaMA-2-7B at 50% sparsity, adaptive allocation beats a uniform per-layer ratio: perplexity drops from 10.02 to 8.30 and average zero-shot accuracy rises from 0.455 to 0.496. GroupMask also outperforms 2:4 and structured baselines on five LLaMA/Qwen models.

Critical strengths
  • A clear question with a controlled comparison. Table 3 shows that adaptive allocation clearly beats uniform allocation when only the per-layer ratio differs. This is the paper’s most informative result.

  • An interpretable, low-cost analysis. The learned masks show consistent patterns: coupled Q/K and Gate/Up budgets, and heavy pruning of deep-layer Q/K. The method needs only about 3.7 GPU-hours with frozen weights.

Critical weaknesses
  1. The central conclusion is confounded, and the method is incremental. The claim that the N:M pattern, rather than adaptive allocation, limits non-uniform sparsity rests on contrasting ProxSparse’s N:M result with Table 3. Those results differ in three factors at once: sparsity pattern (N:M vs. group), allocation method (heuristic vs. learned), and allocation granularity (per block vs. per projection). Table 3 varies none of these factors, so the gap cannot be attributed to the pattern. The paper’s own finding that projection type explains about half of the budget variance makes granularity a plausible alternative explanation. Without learned per-projection allocation under N:M, or OWL-style allocation under group sparsity, the supported result is only that adaptive allocation helps under group sparsity. That is expected from prior unstructured-pruning work. The method itself (GRU hypernetwork, Gumbel-Sigmoid/STE, frozen weights, distillation, global budget) largely follows DISP-LLM.

  2. The baselines omit the method’s direct competitors, so no practical advantage over learned N:M sparsity is shown. GroupMask is a learned mask method, yet it is compared only with one-shot 2:4 methods (SparseGPT, Wanda; perplexity only, absent from the zero-shot table) and structured methods (LLM-Pruner, SliceGPT, MoDeGPT, DISP-LLM). Both comparisons favor GroupMask: the former require no training, and the latter produce dense models with real speedups but are compared at equal parameter count rather than equal inference cost. Learnable N:M methods are absent from the comparison. The authors note that MaskLLM is excluded because of its heavy computational cost; however, ProxSparse is far less computationally intensive, and its N:M observation is the paper’s own motivation. Why isn’t it included in the comparison?

Questions for authors
  1. Isolating the cause of the gain. The conclusion that the N:M pattern limits adaptive allocation is drawn by contrasting results that also differ in allocation method (heuristic vs. learned) and granularity (per block vs. per projection). Could you add one or both of these controls?
  • learned per-projection allocation under N:M (e.g., per-projection N with fixed M under a global budget);
  • OWL-style per-block allocation under group sparsity.

If neither is feasible, would you consider framing the N:M pattern restriction as a hypothesis rather than a definitive conclusion in the abstract and introduction?

  1. Comparison with learned N:M masks. MaskLLM is excluded due to its computational cost, but ProxSparse is considerably cheaper and is the source of the N:M observation motivating this work. Could you compare against ProxSparse, or another learnable N:M method, at a matched calibration budget? Without this comparison, it is unclear whether group sparsity offers any accuracy or efficiency advantage over learned 2:4 masks, which have native hardware support.

  2. Novelty relative to DISP-LLM. The hypernetwork design, Gumbel-Sigmoid/STE, frozen weights, distillation, and global budget regularizer closely follow DISP-LLM. Beyond changing the pruning unit to weight groups, what technical challenges did group-level masks introduce, and how does GroupMask address them? How does it relate to HyperPrune (Sun & Sakuma, ICLR 2026), which also generates semi-structured LLM masks with a hypernetwork?

  3. Practical efficiency and robustness. Could you report end-to-end bf16 latency and memory, including mask or index overhead, for the evaluated 1256 configuration? The reported 1.09× speedup is kernel-level, in FP32, at group size 64. Is the adaptive-vs-uniform gap in Table 3 consistent across seeds and on at least one other model?

5 comments

We thank the reviewer for a precise diagnosis. The confound analysis was right, and the revision follows it: we ran the suggested control, withdrew the conclusion it did not support, and now state the narrower result that the evidence supports. Changes are marked in blue; page numbers refer to the revised PDF.

Q1 / W1. Isolating the cause of the gain.

We added the OWL-style per-block control the reviewer proposed, plus a learned per-block variant. The group pattern, sparsity, training budget and calibration data are the same for all rows:

Allocation (LLaMA-2-7B, 1×64, 50%)GranularityPPL ↓Zero-shot ↑
Uniform–9.11 ± .020.483 ± .003
OWL (heuristic)per block9.09 ± .080.485 ± .002
Learnedper block8.86 ± .100.500 ± .001
Learned (GroupMask)per projection8.21 ± .040.514 ± .008

Source: Sec. 3.4 and Table 3 (p. 6–7). ± is the standard deviation over three seeds. All rows export at 0.49984–0.50014 sparsity (App. B, last paragraph, p. 16).

  • The result supports the reviewer's alternative explanation. Per-block allocation, heuristic or learned, recovers at most 0.25 PPL. Per-projection allocation recovers 0.90.
  • On zero-shot accuracy, learning and granularity contribute comparably (Sec. 3.4, p. 6).
  • We did not run learned per-projection allocation under N:M. It needs a mask parameterization over mixed N:M patterns and matching kernels, which we could not build reliably in the rebuttal period. We therefore removed the claim that the N:M pattern limits adaptive allocation. The Abstract, Sec. 1 (p. 2; "This leaves an open question …"), the contributions (p. 2) and Sec. 5 (p. 10) now report only what the controls show.

We think the finding still goes beyond prior unstructured work. OWL assigns one ratio per Transformer block and reports that per-layer assignment degrades performance (Sec. 4, p. 10). Under group sparsity we find the opposite: the per-block budget barely helps, and the gain comes from letting projections within a block differ. The learned budgets are strongly non-uniform:

Fraction pruned at 50% globalQKVOGateUpDown
LLaMA-2-7B (MHA).80.81.52.46.51.49.28

Source: Figure 3 and Table 5 (p. 9); per-model ranges in Table 7 (App. B, p. 16). In GQA models (LLaMA-3, Qwen3), V is pruned by only 9–12% ("Analysis of learned pruning structures", Sec. 3.5, p. 8).

W2 / Q2. Learned N:M baselines, and cost-matched comparison with structured pruning.

MethodPPL 7B ↓PPL 13B ↓Zero-shot 7B ↑Speedup B1·128
ProxSparse 2:4 (official, C4 calib.)8.516.620.607≤ 1.18× (any 2:4 path)
SparseGPT 2:4 (matched calib.)8.136.670.592≤ 1.18×
DISP-LLM (structured, measured)––0.475 (timed ckpt: 0.463)1.33×
GroupMask 1×648.217.260.5141.49×
GroupMask 1×64 + LoRA7.897.030.5191.49×

Sources: Table 1 (p. 6); Figure 4 (p. 9); Table 8 (p. 20).

  • ProxSparse leads on 13B perplexity and on zero-shot accuracy. We state this in Sec. 3.3 (p. 6).
  • ProxSparse is evaluated from its official checkpoints, calibrated on 400 C4 sequences of 4096 tokens (about 1.6M tokens; Table 13, p. 22). We did not re-calibrate it on WikiText, so its 7B WikiText perplexity is not a matched comparison. With C4 calibration for both, ProxSparse leads GroupMask on LLaMA-2-7B: 8.51 vs. 9.88 PPL and 0.607 vs. 0.552 zero-shot ("Group granularity and calibration data", Sec. 3.5, p. 8; Table 8, p. 20). GroupMask uses 81.92M token presentations.
  • For SparseGPT and Wanda we did align calibration with GroupMask. 16× more calibration data changes them by at most 0.07 PPL (App. G, last paragraph, p. 22).
  • Equal inference cost. We timed DISP-LLM from our reproduced checkpoint with its own per-layer widths (Sec. 3.6, last paragraph, p. 9). At measured cost, GroupMask is both faster and more accurate than this structured baseline.

Q3. Novelty relative to DISP-LLM, and HyperPrune. We agree that GroupMask follows DISP-LLM's optimization line, and Sec. 4 (p. 10) now says so. Moving to weight groups raised three concrete problems:

Challenge with group masksHow GroupMask addresses itEvidence
~101M groups in LLaMA-2-7B at 1×64. A DISP-style head (128→128\rightarrow #groups) would need ~13B parameters.Factorized heads 128→8→128\rightarrow8\rightarrow #groups: 911M parameters, discarded after export, train on one B200 (39.4 GB)Sec. 2.2, p. 3; Table 6, p. 16
Is the hypernetwork needed, or just a parameterization?Direct group logits: 11.19 PPL vs. 8.21. With the same near-dense init, they collapse (> 9000).Table 4, p. 8; App. D, p. 18–19
Gumbel noise makes the trained masks differ from the exported maskDeterministic hard gates + STE: export applies the training rule. Budget met within 1.6×10−41.6\times10^{-4} (Table 3 models); no projection emptied.Sec. 2.2, p. 3; App. B, p. 16; App. H, p. 22
Group masks are useful only if they execute fastCompressed 1×64 format (51.6% of dense storage, including indices) and decoding kernelsSec. 3.6, p. 8–9; App. C, p. 16–17

HyperPrune (Sun & Sakuma, ICLR 2026) is now cited (Sec. 4, p. 10). It also shares one hypernetwork across layers, but it selects N:M patterns layer by layer with a reconstruction objective, so every projection keeps the same local ratio. GroupMask learns all projections jointly under one global budget with end-to-end distillation, which is what allows budgets to differ across projections.

Q4. End-to-end efficiency. These are BF16 full-request measurements (prefill plus 128 generated tokens) using the actual exported masks.

Speedup over dense, RTX PRO 6000B1·128B1·2048B8·128B8·2048Peak mem.
Fastest 2:4 path1.18×1.16×1.02×1.02×8.21 GB (cuSPARSELt)
GroupMask 1×641.49×1.32×1.31×1.13×7.82 GB

Sources: Figure 4 (p. 9) and Sec. 3.6 (p. 8–9); Table 8 (p. 20). Dense uses 13.86 GB. Peak memory includes the index overhead. With prefill also captured in a CUDA graph, cuSPARSELt's per-call overhead at prompt 128 disappears (103 → 14 ms) and the fastest 2:4 path reaches 1.25× / 1.18× / 1.06× / 1.03×, versus 1.56× / 1.35× / 1.33× / 1.16× for GroupMask in the same run; the ranking is unchanged (App. C, p. 18).

The reviewer asked about the 1×256 configuration specifically. We measured it with the same protocol on an A100 and an RTX 5060 Ti:

Full request, batch one1×256 (original)1×1281×64 (new main)1×32
A100, prompt 128 → 128 tokens1.41×1.40×1.36×1.32×
A100, all prompts1.31–1.44×
RTX 5060 Ti, all prompts1.52–1.77×
Peak memory (A100, prompt 128)7.58 GB7.82 GB

Source: App. C, "Group width" paragraph (p. 17). Dense: 13.86 GB.

Speed barely changes down to 1×64, while 1×64 is more accurate: 8.21 PPL, versus 8.69 for the original 1×256 checkpoint (App. F, p. 21). We therefore made 1×64 the main configuration.

Limitations (Sec. 5, p. 10): prefill is not accelerated, and on A100 and B200 GroupMask is slower than dense at batch eight.

Q4. Robustness across seeds and models.

Adaptive vs. uniform (three seeds each)PPL ↓Zero-shot ↑
LLaMA-2-7B8.21 ± 0.04 vs. 9.11 ± 0.020.514 ± 0.008 vs. 0.483 ± 0.003
LLaMA-3-8B12.18 ± 0.08 vs. 13.84 ± 0.090.505 ± 0.005 vs. 0.466 ± 0.002

Source: Table 3 and Sec. 3.4 (p. 6–7).

We hope the revision addresses the reviewer's concerns. We are grateful for the confound analysis, which made the paper's claims more accurate.

I thank the authors for the extensive rebuttal and the new experiments. The added allocation controls (Table 3) resolve my confound concern, and I appreciate that the N:M claim has been withdrawn; the evidence now supports that per-projection granularity, rather than the sparsity pattern, drives the allocation gain. However, the rebuttal also confirms my remaining concerns about significance. The method follows DISP-LLM, and the group-specific additions are largely engineering. With fair baselines, GroupMask no longer outperforms 2:4 methods: zero-shot accuracy is 6-9 points lower, and under matched C4 calibration ProxSparse is better in perplexity. The remaining advantage is decoding speed at batch size one, compared at equal sparsity rather than equal accuracy. Comparisons with MaskLLM, a calibration-matched ProxSparse, and learned allocation under N:M are still missing. Finally, the revision changes the main configuration, the gating mechanism, the evaluation protocol and the central claim; this amounts to a substantially different paper that would benefit from a full new round of review. I therefore keep my score of 2.

Reviewer_tdRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author proposes GroupMask, a pruning method that removes regular groups of weights while allowing sparsity to vary across individual linear projections under a global budget. A bidirectional-GRU hypernetwork generates group masks, which are optimized through a Gumbel-Sigmoid relaxation, sparsity regularization, and self-distillation while keeping pretrained weights frozen. Experiments cover five LLaMA and Qwen models from 7B to 14B, with ablations on allocation, group granularity, and the hypernetwork. The paper reports improvements over uniform group-wise allocation and the evaluated pruning baselines, alongside preliminary kernel-level speedups.

Critical strengths

S1: The adaptive-versus-uniform comparison directly supports the method’s core benefit

S2: The evaluation extends beyond one model and provides interpretable allocation patterns

Critical weaknesses

W1: The comparison with learned semi-structured pruning is incomplete. Table 1 includes only one-shot SparseGPT and Wanda as 2:4 baselines, Table 2 omits 2:4 baselines entirely, and neither MaskLLM nor ProxSparse—both discussed in the paper—is evaluated. The uniform-group ablation establishes a benefit within GroupMask’s pattern, but does not separate the effect of changing the pattern from learning the allocation or assigning budgets per projection rather than per Transformer block, as Section 3.4 acknowledges. Comparisons with learned semi-structured methods under clearly reported calibration and compute budgets are important to establish the contribution beyond the narrower within-pattern result.

W2: The practical accuracy–efficiency tradeoff remains unestablished. Section 3.5 benchmarks only one FP32 gate-projection matrix with N=65,536N=65{,}536, obtaining approximately 1.09–1.11× kernel speedup at the main 50% sparsity setting; the larger 1.88× speedup requires 90% sparsity, where model quality is not evaluated. The authors appropriately acknowledge that end-to-end acceleration remains future work, but without representative inference measurements, it is unclear whether the additional allocation flexibility compensates for giving up native N:M support. Matched quality, latency, and memory measurements would make the practical case substantially stronger.

Questions for authors

Q1: What calibration datasets, sample counts, and optimization budgets were used for each baseline in Tables 1–2? Are there existing comparisons with ProxSparse or a compute-limited MaskLLM configuration?

Q2: What are the final exported sparsities for the adaptive, uniform, and no-hypernetwork variants? Figure 5a shows nonzero regularization loss for the no-hypernetwork variant at the end of training, so please clarify whether Table 4 compares models at the same achieved sparsity. 43616_GroupMask_Layer_Adaptive

Q3: Does Figure 5 use actual exported GroupMask masks or synthetic sparsity patterns, and what exact group shapes are benchmarked? Are any existing BF16 or end-to-end measurements available?

Q4: What is the hypernetwork’s trainable parameter count, and how many GPUs were used for the reported 3.7 GPU-hours? Table 5’s hardware description, “1 × NVIDIA DGX B200,” leaves the GPU count unclear

3 comments

We thank the reviewer for the careful reading and for questions that pointed directly at the experiments the paper was missing. We ran all of them except MaskLLM (see below). Changes are marked in blue; page numbers refer to the revised PDF.

W1 / Q1. Learned 2:4 baselines under matched calibration.

  • The learned 2:4 method ProxSparse is now included, evaluated from its official checkpoints. These are calibrated on C4 (400 sequences of 4096 tokens), and we did not re-calibrate them on WikiText (Table 1 footnote †, p. 6; Table 13, p. 22).
  • SparseGPT and Wanda are re-run with the same WikiText-103 calibration as GroupMask. The original submission cited their C4-calibrated numbers; we reproduce those under C4 (App. G, last paragraph, p. 22).
  • 2:4 zero-shot results now cover all five models. LLaMA-3 and Qwen3 are in Table 9 (App. E, p. 20).
LLaMA-2, 50% sparsityUpd.PPL 7B ↓PPL 13B ↓Zero-shot 7B ↑Zero-shot 13B ↑
Dense–5.124.570.7050.732
SparseGPT 2:4Yes8.136.670.5920.642
Wanda 2:4No10.537.800.5790.639
ProxSparse 2:4No8.516.620.6070.662
GroupMask 1×64No8.217.260.5140.564
GroupMask 1×64 + LoRAYes7.897.030.5190.563

Sources: Table 1 (p. 6) and Table 8 (App. C, p. 20). All rows use WikiText calibration except ProxSparse (official C4-calibrated checkpoints), so its 7B perplexity is not a matched comparison. With C4 calibration for both, ProxSparse leads GroupMask on LLaMA-2-7B: 8.51 vs. 9.88 PPL and 0.607 vs. 0.552 zero-shot ("Group granularity and calibration data", Sec. 3.5, p. 8; Table 8, p. 20).

2:4 methods retain higher zero-shot accuracy. We state this in Sec. 3.3 (p. 6) and Sec. 5 (p. 10), and we removed the submission's claim of the best perplexity. MaskLLM could not be run within the rebuttal period.

Q1. Calibration data, sample counts and budgets for every baseline in Tables 1–2:

MethodCalibrationSamples / budgetUpdates weights
GroupMaskWikiText-10340k × 2048 tokens (81.92M), 40k steps, 4.2–4.5 h on one B200No
GroupMask + LoRAWikiText-103above + 5k steps (10.24M tokens)Masked LoRA
SparseGPT 2:4WikiText-103128 × 4096 tokens (0.52M), one-shotYes
Wanda 2:4WikiText-103128 × 4096 tokens (0.52M), one-shotNo
ProxSparse 2:4C4 (official checkpoint)400 × 4096 tokens; number of passes not reportedNo
LLM-PrunerWikiText-210 × 64 tokens for Taylor scores; no LoRA recoveryNo
SliceGPT, MoDeGPTWikiText-2 / WikiTextofficial code, default settingsYes
DISP-LLMWikiTextofficial code; Table 1 PPL from its paperNo

Sources: Table 13 and App. G (p. 22); Table 6 (p. 16).

To test whether this gap favors us, we raised the one-shot calibration by 16×:

Calibration sequences (tokens)SparseGPT PPL / zero-shotWanda PPL / zero-shot
128 (0.5M), as in Table 18.13 / 0.59210.53 / 0.579
512 (2.1M)8.06 / 0.58410.52 / 0.576
1024 (4.2M)8.09 / 0.59310.50 / 0.578
2048 (8.4M)8.06 / 0.59210.57 / 0.576

Source: App. G, last paragraph (p. 22). The one-shot baselines are saturated: at most 0.07 PPL and 0.8 points of change.

W1. Pattern versus allocation. Sec. 3.4 and Table 3 (p. 6–7) now hold the 1×641\times64 pattern, 50% sparsity and 40k-step budget fixed, and vary only the allocation:

Allocation (LLaMA-2-7B)PPL ↓Zero-shot ↑Exported sparsity
Uniform9.11 ± .020.483 ± .0030.49997–0.49998
OWL, per block9.09 ± .080.485 ± .0020.49998–0.49999
Learned, per block8.86 ± .100.500 ± .0010.49984–0.50004
Learned, per projection (GroupMask)8.21 ± .040.514 ± .0080.49996–0.50014

Sources: Table 3 (p. 7); sparsities from App. B, last paragraph (p. 16). ± is the standard deviation over three seeds.

  • Projection granularity accounts for most of the perplexity gain (0.65 of 0.90).
  • On zero-shot accuracy, learning and granularity contribute comparably.
  • LLaMA-3-8B shows the same direction over three seeds (Sec. 3.4, p. 6).
  • Following this evidence, we withdrew the claim that the N:M pattern itself limits adaptive allocation (Abstract; Sec. 1, p. 2; Sec. 5, p. 10).

W2 / Q3. Practical accuracy–efficiency tradeoff.

The FP32 single-matrix benchmark is replaced by BF16 full-request measurements: prefill plus 128 generated tokens, static KV cache, CUDA Graph, median of five runs on an RTX PRO 6000. All GroupMask timings use the actual exported masks.

Speedup over dense (B · prompt)B1·128B1·2048B8·128B8·2048Peak mem. (B1·128)
Dense1.00×1.00×1.00×1.00×13.86 GB
cuSPARSELt 2:41.08×1.10×1.02×1.02×8.21 GB
Triton 2:4 (tuned)1.14×1.08×0.62×0.75×8.57 GB
Hybrid 2:41.18×1.16×0.63×0.76×–
DISP-LLM (structured)1.33×1.23×1.04×1.01×7.44 GB
GroupMask 1×641.49×1.32×1.31×1.13×7.82 GB

Sources: Figure 4 (p. 9) and Sec. 3.6 (p. 8–9); Table 8 (p. 20); protocol in App. C (p. 16–17). With all real weights, every speedup changes by at most 0.02×, and the compressed model reproduces the masked model exactly (App. C, p. 17). With prefill also captured in a CUDA graph, cuSPARSELt's per-call overhead at prompt 128 disappears (103 → 14 ms) and the fastest 2:4 path reaches 1.25× / 1.18× / 1.06× / 1.03×, versus 1.56× / 1.35× / 1.33× / 1.16× for GroupMask in the same run; the ranking is unchanged (App. C, p. 18).

The original submission's 1.88× needed 90% sparsity, where quality was not evaluated. We now report quality and speed together at higher sparsity:

LLaMA-2-7B, 1×6450%60%70%
PPL ↓8.219.5111.65
Zero-shot ↑0.5140.4540.431
Speedup (B1·128)1.49×1.69×1.87×
Peak memory7.82 GB6.48 GB5.14 GB

Source: Table 5 (p. 9) and "Effect of sparsity ratio", Sec. 3.5 (p. 8). 2:4 hardware cannot express 60% or 70%.

Where GroupMask does not help.

  • Prefill is not accelerated.
  • On A100 and B200, GroupMask is slower than dense at batch eight. On B200 it runs 1.21× / 1.17× at batch one but 0.68–0.81× at batch eight (App. C, p. 17).
  • At 50% sparsity, 2:4 is more accurate. Our speedups are compared at equal sparsity, not equal accuracy (Sec. 5, p. 10).

Q2. Achieved sparsity. All compared variants export at 50% (table under W1). The original Figure 5a used the earlier noisy soft-gate protocol. All reported runs now train with the same deterministic hard-gate/STE rule, and GroupMask exports with that same threshold. The no-hypernetwork control is exported by keeping its top 50% logits (0.500001); its training hard gates already keep exactly 50.0000%, so the exported mask matches the training mask up to a 10⁻⁶ fraction of groups (Sec. 2.2, p. 3; App. B, p. 16; App. H, p. 22). The training curves are now Figure 8 (App. D, p. 19). We also checked initialization:

LLaMA-2-7B, 1×64, WikiTextGate init.PPL ↓Zero-shot ↑
GroupMask (hypernetwork)near-dense (b = 3)8.210.514
Direct logits (Table 4)random 50% (b = 0)11.190.455
Direct logitsnear-dense (b = 3)> 90000.371

Sources: Table 4 (p. 8); App. D, last sentence (p. 19).

Q4. Parameters and hardware.

ItemValueWhere
Trainable parameters (1×64, 7B)910,960,896, nine per group; discarded after exportSec. 2.2, p. 3; Table 6, p. 16
GPUsone B200, single processTable 6, p. 16
Training time (LLaMA-2-7B)4.2–4.5 hTable 6, p. 16
Peak allocated memory (100-step probe, full config.)39.38 GBTable 6, p. 16

The ambiguous "1 × NVIDIA DGX B200" has been corrected.

We hope these additions address the reviewer's concerns, and we would be glad to clarify anything further.

Dear Reviewer tdRs,

We thank you again for the thoughtful and constructive feedback. We have carefully considered the concerns raised in the review and provided detailed responses, clarifications, and additional evidence where applicable. As the discussion period is approaching its end, we would appreciate any further feedback on whether our responses adequately address your main concerns, or if any questions remain. We would be happy to provide further clarification or make necessary revisions during the remaining discussion period.

Kind Regards, The Authors

Comments on the paper

3 comments

We thank all three reviewers for detailed, constructive reviews. They converged on three requests: separate the cause of the allocation gain, compare fairly with learned 2:4 methods, and show real inference benefit. We ran every requested experiment we could complete in the rebuttal period, and we changed our claims where the new evidence required it. Changes are marked in blue; in tables, a blue row label marks a new or re-measured row. Page numbers refer to the revised PDF.

What did not change. The problem, the method framework, the models and the core finding are the same as in the submission:

  • Framework: a shared hypernetwork generates binary group selectors under a global sparsity budget, trained by self-distillation with frozen weights.
  • Models: five LLaMA and Qwen models from 7B to 14B.
  • Core finding: under group sparsity, learned adaptive allocation outperforms uniform allocation. The submission showed this for 1×2561\times256 (its Table 3). The revision confirms it for 1×641\times64 with three seeds and on a second model family (Table 3 and Sec. 3.4, p. 6–7).

What changed, and why. Every change traces to a reviewer request:

ChangeRequested byWhere
Allocation controls with the group pattern fixed: uniform, OWL per-block, learned per-block, learned per-projectionyPS7 Q1, tdRs W1Sec. 3.4, Table 3, p. 6–7
Claim that "the N:M pattern limits adaptive allocation" withdrawn; the supported result is stated insteadyPS7 Q1Abstract; Sec. 1, p. 2; Sec. 5, p. 10
Learned 2:4 baseline ProxSparse addedall threeTable 1, p. 6; Fig. 4(b), p. 9; Table 8, p. 20
2:4 zero-shot results for all five modelstdRs W1, yPS7 W2Fig. 4(b); Table 8; Table 9, App. E, p. 20
SparseGPT and Wanda re-run with GroupMask's calibration data; C4 citations reproduced; calibration budget sweeptdRs Q1, hBd8 Q2Sec. 3.1, p. 5; App. G, p. 21–22; Table 13, p. 22
BF16 full-request latency and memory against dense, three 2:4 execution paths and structured DISP-LLMall threeFig. 4, p. 9; Sec. 3.6, p. 8–9; Table 8, p. 20; App. C, p. 16–18
Speed and quality together at 60% and 70% sparsitytdRs W2Table 5, p. 9; Sec. 3.5, p. 8
Three seeds for adaptive and uniform, and a second model family (LLaMA-3-8B)yPS7 Q4, hBd8 Q3Table 3 and Sec. 3.4, p. 6–7
Exported sparsity of every compared varianttdRs Q2App. B, p. 16
Hypernetwork parameter count, hardware and training cost; initialization check of the no-hypernetwork controltdRs Q2, Q4Sec. 2.2, p. 3; Table 6, p. 16; App. D, p. 18–19
DISP-LLM relationship stated; HyperPrune citedyPS7 Q3Sec. 4, p. 10

Three implementation changes, each made to answer a request above:

  1. Deterministic hard gates. Gumbel noise is no longer injected during training. Training now uses the same deterministic rule as the export, which makes the achieved sparsity exact (Sec. 2.2, p. 3; App. H, p. 22).
  2. Bottleneck output heads. This is a memory optimization that lets the finer groups needed for the requested group-shape and speed studies fit on one GPU (Sec. 2.2, p. 3).
  3. Main configuration 1×641\times64. It is the best point of the group-shape study once that study is extended to finer shapes (Table 4, p. 8), and it is the configuration we measure end to end. All original 1×2561\times256 results remain in App. F.

We also unified the perplexity protocol to non-overlapping 4096-token windows. Re-measuring the original 1×2561\times256 checkpoint under this protocol gives 8.69 instead of the 8.30 reported in the submission. The difference comes from the evaluation window alone, not from the model (App. F, p. 21).

How our view of the method changed. The new experiments sharpened our understanding of where GroupMask's value lies. It does not lie in beating 2:4 on accuracy, which it does not do at 50% sparsity. It lies in combining a sparse format that decodes fast with projection-specific allocation that recovers quality. Comment 3/3 states this framing and its costs explicitly.

1. Where the allocation gain comes from. We held the 1×641\times64 group pattern, 50% sparsity and training budget fixed, and varied only the allocation (Table 3, p. 7):

AllocationPPL
Uniform9.11 ± 0.02
OWL per-block9.09 ± 0.08
Learned per-block8.86 ± 0.10
Learned per-projection8.21 ± 0.04
  • Projection-level granularity explains most of the perplexity gain. On zero-shot accuracy, learning and granularity contribute comparably.
  • The gap is far larger than seed variation (± is over three seeds).
  • It holds on LLaMA-3-8B: 12.18 ± 0.08 vs. 13.84 ± 0.09.

2. Comparison with 2:4, including results that do not favor us.

  • With matched calibration, perplexity is on par on LLaMA-2-7B: SparseGPT 8.13, GroupMask 8.21 (7.89 with masked LoRA). SparseGPT and ProxSparse lead on 13B (Table 1, p. 6).
  • 2:4 methods retain higher zero-shot accuracy: 0.579–0.607 versus 0.514 on LLaMA-2-7B (Sec. 3.3, p. 6). We removed the submission's claim of the best perplexity.
  • The one-shot baselines are not limited by calibration volume: 16× more calibration data changes them by at most 0.07 PPL (App. G, p. 22).
  • These comparisons are not under equal constraints. GroupMask keeps every pretrained weight frozen and decides whole 64-weight groups, whereas 2:4 chooses 2 of every 4 weights and SparseGPT also reconstructs the retained weights. The like-for-like comparison is Wanda, which is also mask-only and uses the same WikiText calibration:
Mask-only, same calibrationSelection unitPPL 7B ↓PPL 13B ↓
Wanda 2:42 of 4 weights10.537.80
GroupMask 1×6464-weight group8.217.26

With 16× coarser selection units, GroupMask is 2.3 PPL lower on 7B and 0.5 lower on 13B (Table 1, p. 6). Once GroupMask may also adapt the retained weights (masked LoRA), it reaches 7.89, below SparseGPT's 8.13. This advantage is in perplexity only: Wanda's zero-shot average remains higher (0.579 vs. 0.514).

Goal. A sparse LLM that runs faster and uses less memory in low-batch, memory-bound decoding, while each projection keeps the share of weights it needs. Two design constraints define this problem, and one optimization choice solves it:

ChoiceWhat it buysWhere
Constraint: contiguous 64-weight groups, not 2-of-4Bandwidth-efficient decoding: contiguous 128-byte segments with one index per 64 values. 1.13–1.49× dense, vs. at most 1.18× for 2:4 on the same GPU (cuSPARSELt, our tuned Triton kernel, or their hybrid)Sec. 3.6, Fig. 4, p. 8–9
Constraint: one global budget, not a fixed local ratioBudgets vary by projection (Q/K about 80% pruned, down 28%), which drives the quality gain. Any sparsity is possible, e.g. 1.69× / 1.87× at 60% / 70%Table 3, p. 7; Fig. 3, Table 5, p. 9
Optimization choice: frozen weights, mask only (not needed for speed)The compressed model is an exact sub-network of the original, and allocation effects are measured without weight updates confounding them. Masked LoRA stays optional and merges without changing the format.Sec. 1, p. 2; Sec. 2.4, p. 5; Table 3, p. 7

The cost. These are trade-offs, not free gains. At 50% sparsity, 2:4 remains more accurate. The gap does not come mainly from freezing the weights: across the ten model/calibration settings of Table 2 (p. 7), masked LoRA raises zero-shot accuracy by at most 1.5 points, far from the 6–9-point gap. Projection-specific allocation works in the other direction: at a fixed format, it recovers quality (Table 3).

Real inference benefit (BF16, prefill plus 128 generated tokens, RTX PRO 6000; Fig. 4, p. 9; Table 8, p. 20):

MethodSpeedup over densePeak memory
GroupMask1.13–1.49×7.82 GB
Fastest 2:4 path1.02–1.18×8.21 GB (cuSPARSELt)
Structured DISP-LLM1.01–1.33×7.44 GB
Dense1.00×13.86 GB

With prefill also graph-captured, the fastest 2:4 path reaches at most 1.25× (GroupMask 1.56× in the same run; App. C, p. 18). DISP-LLM is also less accurate (0.475 in Table 2, 0.463 for the timed checkpoint, vs. 0.514 zero-shot). Speedups change by at most 0.02× with real weights, and the compressed model reproduces the masked model exactly (App. C, p. 17).

Limitations (Sec. 5, p. 10). Prefill is not accelerated. GroupMask is slower than dense at batch eight on A100 and B200. 2:4 is more accurate at 50% sparsity, and our speedups are compared at equal sparsity, not equal accuracy. We could not evaluate MaskLLM within the rebuttal period.

Summary. GroupMask trades some accuracy for a sparse representation that delivers measurable decoding speedups and memory savings, while recovering quality through projection-specific allocation under a global sparsity budget. It is faster than 2:4 in low-batch decoding, more accurate than structured pruning in zero-shot evaluation on all five models, and reaches sparsities that 2:4 cannot express.

Detailed answers to each reviewer follow in the individual responses. We thank the reviewers again; their questions made the paper's claims more precise and better supported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.