- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GroupMask, a layer-adaptive group-wise sparsification method for pruning pretrained LLMs. It learns group masks and allocates a global sparsity budget across layers and projections, aiming to preserve more important parameters than uniform pruning. The method uses differentiable mask learning while keeping the pretrained weights frozen. Experiments on several LLM families evaluate perplexity, downstream-task accuracy, and different sparsity configurations. The main claim is that adaptive group-wise sparsity improves pruning quality.
Critical strengths
-
The paper studies an important limitation of uniform pruning. A global sparsity budget does not need to be distributed equally across layers and projections. Learning this allocation is a reasonable way to preserve parameters where they are more useful.
-
The adaptive-versus-uniform ablation provides useful evidence. The comparison uses the same group structure, calibration data, sparsity target, and training schedule for the tested LLaMA-2 7B setting. This helps isolate the contribution of adaptive allocation.
-
The evaluation includes multiple model families and ablations. The experiments cover LLaMA and Qwen models, with several sparsity settings and analyses of group shape and calibration data. This provides useful evidence about the method’s behavior beyond a single configuration.
Critical weaknesses
-
The practical inference benefit is not sufficiently demonstrated. The main experiments use a 1×256 group structure, which is different from the standard 2:4 pattern commonly supported by sparse inference hardware. The reported kernel-level speedup is only around 1.09× at 50% sparsity, and end-to-end model latency and throughput are not sufficiently established. Therefore, the results mainly demonstrate pruning quality, not a clear deployment advantage.
-
Comparisons with existing methods do not consistently match deployment costs and experimental settings. Equal parameter sparsity does not imply equal FLOPs or inference latency when the sparsity structures differ. Some comparisons also use different calibration recipes. These differences make it difficult to determine whether the reported gains reflect better allocation or differences in the experimental setup.
-
The comparison with learned-mask methods is incomplete, and the generality of the allocation gains remains uncertain. MaskLLM is omitted from the main comparison because of its higher training cost. Although this cost is relevant, a quality-versus-training-cost comparison would be more informative. In addition, the controlled adaptive-versus-uniform evidence is concentrated on LLaMA-2 7B, and variability across independent mask-learning runs is not sufficiently established.
Questions for authors
-
Can you report end-to-end inference latency and throughput for exported GroupMask models, compared with standard 2:4 pruning methods at matched hardware, batch size, and model quality or measured latency?
-
Can you provide comparisons with relevant pruning baselines using the same calibration data and clearly matched sparsity or compute budgets, and include a quality-versus-training-cost comparison with MaskLLM or another learned-mask method?
-
Does the adaptive-versus-uniform improvement remain consistent across additional model families and independent mask-learning runs? Can you report the variability of the main results across seeds?
6 comments
W2 / Q2. Matched calibration, cost and learned-mask baselines.
Deployment cost. We now compare measured latency and memory, not parameter counts (table above). This includes a structured baseline timed with its own architecture (Table 8, footnote §, p. 20).
Calibration. SparseGPT and Wanda are re-run with the same WikiText-103 data as GroupMask (Sec. 3.1, p. 5). The original submission cited C4-calibrated numbers, which we reproduce under C4 (App. G, last paragraph, p. 22).
| LLaMA-2, 50% (WikiText calibration unless noted) | PPL 7B ↓ | PPL 13B ↓ | Zero-shot 7B ↑ |
|---|---|---|---|
| SparseGPT 2:4 | 8.13 | 6.67 | 0.592 |
| Wanda 2:4 | 10.53 | 7.80 | 0.579 |
| ProxSparse 2:4 (learned, official C4 checkpoint) | 8.51 | 6.62 | 0.607 |
| GroupMask 1×64 | 8.21 | 7.26 | 0.514 |
| GroupMask 1×64 + LoRA | 7.89 | 7.03 | 0.519 |
Sources: Table 1 (p. 6); Table 8 (p. 20). We removed the submission's claim of the best perplexity. ProxSparse is evaluated from its official checkpoints, calibrated on C4 (400 × 4096 tokens), and we did not re-calibrate it on WikiText. With C4 calibration for both, ProxSparse leads GroupMask on LLaMA-2-7B: 8.51 vs. 9.88 PPL and 0.607 vs. 0.552 zero-shot ("Group granularity and calibration data", Sec. 3.5, p. 8; Table 8, p. 20).
Training cost and calibration volume.
| Method | Token presentations | Hardware / time |
|---|---|---|
| GroupMask | 81.92M | 1 B200, 4.2–4.5 h |
| SparseGPT, Wanda | 0.52M | one-shot |
Sources: Table 13 (p. 22); Table 6 (p. 16).
To check whether this difference drives the comparison, we gave the one-shot methods 16× more calibration data:
| Calibration sequences (tokens) | SparseGPT PPL / zero-shot | Wanda PPL / zero-shot |
|---|---|---|
| 128 (0.5M) | 8.13 / 0.592 | 10.53 / 0.579 |
| 512 (2.1M) | 8.06 / 0.584 | 10.52 / 0.576 |
| 1024 (4.2M) | 8.09 / 0.593 | 10.50 / 0.578 |
| 2048 (8.4M) | 8.06 / 0.592 | 10.57 / 0.576 |
Source: App. G, last paragraph (p. 22). They are saturated.
We could not run MaskLLM, or a full quality-versus-training-cost curve, within the rebuttal period, and we do not claim a comparison with it.
W3 / Q3. Generality and run-to-run variability.
| Adaptive vs. uniform (independent runs) | Seeds | PPL ↓ | Zero-shot ↑ |
|---|---|---|---|
| LLaMA-2-7B | 3 + 3 | 8.21 ± 0.04 vs. 9.11 ± 0.02 | 0.514 ± 0.008 vs. 0.483 ± 0.003 |
| LLaMA-3-8B (GQA) | 3 + 3 | 12.18 ± 0.08 vs. 13.84 ± 0.09 | 0.505 ± 0.005 vs. 0.466 ± 0.002 |
| LLaMA-2-7B, 8×8 groups | 3 | 8.15 ± 0.02 | 0.521 ± 0.002 |
Sources: Table 3 and Sec. 3.4 (p. 6–7); Table 4 caption (p. 8).
- On LLaMA-2-7B, the perplexity gap is about 20× the seed standard deviation.
- On LLaMA-3-8B, the gap holds even though the uniform models keep slightly more parameters.
- With the group pattern fixed, uniform, OWL per-block, learned per-block and learned per-projection allocation reach 9.11, 9.09, 8.86 and 8.21 PPL (three seeds each) (Table 3, p. 7). The gain therefore comes mainly from per-projection budgets.
- All LLaMA-2-7B variants in these comparisons export within of 50% sparsity (App. B, last paragraph, p. 16).
We hope these additions address the reviewer's concerns about practicality and fairness, and we would be glad to provide further details.
Dear Reviewer hBd8,
We thank you again for the thoughtful and constructive feedback. We have carefully considered the concerns raised in the review and provided detailed responses, clarifications, and additional evidence where applicable. As the discussion period is approaching its end, we would appreciate any further feedback on whether our responses adequately address your main concerns, or if any questions remain. We would be happy to provide further clarification or make necessary revisions during the remaining discussion period.
Kind Regards, The Authors
We thank the reviewer for focusing on deployment, cost matching and robustness. All three points led to new experiments, and we report what we found, including where it does not favor GroupMask. Changes are marked in blue; page numbers refer to the revised PDF.
W1 / Q1. End-to-end inference against 2:4 at matched hardware and batch size.
All runs are BF16 full requests (prefill plus 128 generated tokens) with the learned GroupMask masks: static KV cache, CUDA Graph decoding, median of five runs, same RTX PRO 6000.
Sources: Figure 4 (p. 9) and Sec. 3.6 (p. 8–9); Table 8 (p. 20); protocol in App. C (p. 16–17).
Matched model quality. At 50% sparsity the 2:4 methods are more accurate, so GroupMask is not faster than 2:4 at equal quality. What it offers is a different point on the tradeoff:
Sources: Figure 4(b), Table 8 (p. 20); Table 5 (p. 9) and "Effect of sparsity ratio", Sec. 3.5 (p. 8).
Limits (Sec. 5, p. 10; App. C, p. 17):