- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SparRotQuant, a framework that jointly optimizes rotation-based quantization and 2:4 semi-structured sparsity for LLM compression. It addresses the incompatibility between quantization-oriented rotations and stable sparse patterns by introducing Probability-aware Balanced Group Separation (PBGS), which combines retention probabilities and Hessian-aware sensitivity to guide differentiable mask learning. The framework applies exact 2:4 pruning to five selected linear projections alongside low-bit quantization. Experiments on LLaMA2, LLaMA3, and Qwen3 under W4A4KV16 and W4A4KV4 demonstrate improved perplexity and zero-shot accuracy over sparse quantization baselines, alongside inference speedups and memory savings.
Critical strengths
-
Joint optimization of quantization and structured sparsity. SparRotQuant introduces PBGS to encourage stable 2:4 sparsity patterns while jointly optimizing rotation matrices, differentiable pruning masks, and quantization parameters. This provides a possible approach to addressing the incompatibility between rotation-based quantization and structured pruning, potentially improving the quality of jointly compressed LLMs.
-
Comprehensive experimental validation with practical efficiency gains. Experiments across multiple LLM families demonstrate consistent improvements in perplexity and zero-shot accuracy over existing sparse quantization baselines. The ablation studies further support the effectiveness of PBGS, while hardware-level and end-to-end evaluations demonstrate meaningful inference acceleration and memory savings.
Critical weaknesses
-
Insufficient novelty and missing closely related work. The paper overlooks DenoiseRotator[1] and RotPruner[2], which similarly employs learnable rotations to concentrate weight importance and improve pruning robustness. Despite the substantial conceptual overlap, the authors neither discuss this closely related work nor provide experimental comparisons. This raises concerns about the novelty of SparRotQuant and whether PBGS offers meaningful improvements beyond existing rotation-based pruning methods.
-
Limited pruning scope and insufficient experimental justification. SparRotQuant applies 2:4 pruning to only five linear projections, excluding W_o and W_down, whereas DenoiseRotator supports pruning all seven projections. Although the authors discuss coupled approximation errors, their analysis does not sufficiently justify excluding these important layers. Furthermore, the rotation-placement ablations involve different amounts of pruned parameters, making the comparisons potentially confounded. The lack of experiments with all seven projections and matched model-wide sparsity raises concerns about the generality and actual compression effectiveness of the proposed method.
-
Typos
- Line 391: "SparRotQuant calibration deployment workflow" -> "SparRotQuant calibration and deployment workflow"
[1] DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration. NeurIPS 2025
[2] RotPruner: Large Language Model Pruning in Rotated Space. https://openreview.net/forum?id=wV9iMiyQcc
Questions for authors
-
Novelty relative to Existing Works. DenoiseRotator and RotPruner similarly employ learnable rotations to concentrate weight importance and improve pruning robustness. Why is this closely related work not discussed? Can the authors clarify the fundamental differences between PBGS and DenoiseRotator's importance concentration objective, particularly regarding their methodological novelty?
-
Comparison with DenoiseRotator. Can the authors provide experimental comparisons with DenoiseRotator under matched quantization settings, calibration protocols, and pruning budgets? Such comparisons would help establish whether SparRotQuant offers meaningful improvements over existing rotation-based pruning methods.
-
Limited pruning scope and experimental fairness. Why does SparRotQuant exclude and from 2:4 pruning, whereas DenoiseRotator supports all seven linear projections? Can the authors evaluate sparsification of these additional layers and report model-wide sparsity ratios? Furthermore, can they provide rotation-placement ablations under matched overall sparsity budgets to distinguish the effects of coupled approximation errors from those of pruning more parameters?
Thank you for the comprehensive response, which addresses most of my concerns. I will keep my original score.