- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Prune-Quantize-Prune (PQP), a joint compression framework that exploits quantization-induced "free sparsity" under symmetric quantization. PQP consists of quantization-aware mask selection, quantization, and grid-constrained compensation to satisfy the target sparsity pattern while minimizing reconstruction error. Experiments across LLaMA and Mistral models demonstrate improvements over existing pruning-quantization baselines.
Critical strengths
- S1. Interesting motivation. Exploiting quantization-induced free sparsity is an intuitive and interesting approach to joint pruning and quantization
Critical weaknesses
-
W1. Limited empirical improvements. While PQP generally outperforms OBR, the improvements are relatively minor in several settings. For instance, under W8A8KV16 quantization (Table 3), PQP improves average accuracy by only 0.23%p and perplexity by 0.02. Such differences may fall within experimental variability. Moreover, Grid-constrained Compensation yields negligible reconstruction error reductions in several layers (Table 1), raising questions about its practical contribution. Also, proposed PQP is impelmented by adopting Hadamard transformation, and it is not clear whether the method itself is benefiting from rotation, not from the proposed method
-
W2. Insufficient methodological clarity and experimental justification. Several implementation details are unclear, including trial mask construction, the distinction between trial and actual quantization, the role of Hadamard rotation, efficiency evaluations. The experimental evaluation also lacks comparisons with relevant methods such as JSQ, despite discussing them in the background.
Questions for authors
Q1. Given the marginal improvements over OBR in several settings, could the authors provide evidence of statistical significance or robustness across calibration seeds?
Q2. Table 1 shows negligible improvements from Grid-constrained Compensation in several layers. How does this translate to end-to-end perplexity and accuracy improvements?
Q3. In Table 4, does "Quantization-aware mask" exclude Grid-constrained Compensation? Does the final row include both components, or compensation alone?
Q4. Why is JSQ excluded from the experimental comparisons despite being discussed in Section 2.3?
Q5. How does Hadamard rotation affect PQP? Could the authors provide results without rotation?
Q6. Could the authors clarify the kernel backends, matrix dimensions, and workloads used for the speedup evaluation in Section 5.5?
6 comments
Response to Reviewer e9j3 (Part 2/5)
Q2: "Table 1 shows negligible improvements from Grid-constrained Compensation in several layers. How does this translate to end-to-end perplexity and accuracy improvements?"
Thank you for raising this question. We analyze calibration reconstruction error for all seven linear projections (Q, K, V, O, Gate, Up, and Down) in every Transformer layer: 224 projections in Llama2-7B and 280 in Llama2-13B. We compare our Grid-constrained Compensation with direct zeroing under W4A16 quantization and 2:4 sparsity. For each projection , the relative reconstruction error reduction is , where and denote the reconstruction errors under direct zeroing and Grid-constrained Compensation, respectively. As shown in Table C, Grid-constrained Compensation reduces calibration reconstruction error in every measured projection, with a maximum reduction of about 75%. Lower layer-wise reconstruction error across these projections better preserves projection outputs and is expected to benefit downstream tasks.
Table D reports end-to-end ablations on both Llama2-7B and Llama2-13B. On Llama2-7B, adding Grid-constrained Compensation to Quantization-aware Mask Selection reduces WikiText-2 perplexity from 9.84 to 9.77 and improves average zero-shot accuracy from 54.40% to 54.69%. On Llama2-13B, the same addition reduces perplexity from 8.05 to 7.84 and improves average zero-shot accuracy from 59.94% to 60.47%. These results show measurable end-to-end improvements on both models alongside the reconstruction gains.
Table C: Calibration reconstruction error reduction with Grid-constrained Compensation relative to direct zeroing
| Model | Improved projections ↑ | Max. reduction (%) ↑ | Min. reduction (%) ↑ | Median reduction (%) ↑ |
|---|---|---|---|---|
| Llama2-7B | 224/224 | 75.1051 | 0.0199 | 6.4382 |
| Llama2-13B | 280/280 | 71.8593 | 0.0111 | 5.6959 |
Table D: End-to-end ablation of PQP components on Llama2-7B and Llama2-13B under W4A16KV16 quantization with 2:4 sparsity
| Model | Configuration | WikiText-2 PPL ↓ | Avg. accuracy (%) ↑ |
|---|---|---|---|
| Llama2-7B | OBR | 10.38 | 54.12 |
| PQP w/o Grid-constrained Compensation | 9.84 | 54.40 | |
| PQP w/ Grid-constrained Compensation | 9.77 | 54.69 | |
| Llama2-13B | OBR | 8.29 | 59.59 |
| PQP w/o Grid-constrained Compensation | 8.05 | 59.94 | |
| PQP w/ Grid-constrained Compensation | 7.84 | 60.47 |
Q3: "In Table 4, does “Quantization-aware mask” exclude Grid-constrained Compensation? Does the final row include both components, or compensation alone?"
Thank you for requesting this clarification. We explain below how the two rows in Table 4 of the manuscript differ and which components are included in each configuration:
- The “Quantization-aware mask” row includes Quantization-aware Mask Selection but excludes Grid-constrained Compensation. This configuration isolates the contribution of Quantization-aware Mask Selection relative to the OBR baseline. Specifically, after quantization, weights selected for additional pruning to satisfy the 2:4 sparsity constraint are directly set to zero, without any compensation or adjustment to the remaining weights at this stage.
- The final row includes both Quantization-aware Mask Selection and Grid-constrained Compensation. Comparing this row with the “Quantization-aware mask” row isolates the additional contribution of Grid-constrained Compensation. It uses the same mask-selection procedure and additionally compensates for the reconstruction error introduced by final re-pruning by updating the retained weights while keeping them on the quantization grid.
We have revised the manuscript text and Table 4 to make this cumulative relationship explicit.
Response to Reviewer e9j3 (Part 3/5)
Q4: "Why is JSQ excluded from the experimental comparisons despite being discussed in Section 2.3?"
Thank you for pointing out this missing comparison. Our method was developed primarily based on OBR's compensation framework, so we initially focused on OBR as the main baseline and did not include JSQ in the experimental comparisons. We agree that JSQ is a relevant baseline. To evaluate whether our proposed components are also effective within the JSQ pipeline, we have now added a comparison between JSQ and JSQ augmented with these components (Table E).
Our proposed components can also be integrated into JSQ. Specifically, JSQ introduces a sparsity metric that accounts for both pruning importance and quantization sensitivity, together with an activation editor to mitigate outliers (Guo et al., 2024). We integrate our Quantization-aware Mask Selection and Grid-constrained Compensation into this pipeline as follows:
- Quantization-aware Mask Selection. We first construct a standard 2:4 mask using JSQ's importance score and perform trial smoothing and RTN quantization. Within each group of four weights, we count the retained weights that are quantized to zero and use this count as the budget for restoring initially pruned weights, prioritizing those with higher JSQ scores. We then apply the relaxed mask to the original weights and repeat smoothing and quantization, allowing quantization-induced zeros to contribute to the target sparsity.
- Grid-constrained Compensation. After actual quantization, we re-prune groups with more than two nonzero weights. For each weight removed during final re-pruning, we use our proposed method to compute the compensation gain for each candidate retained weight. We then select the candidate with the largest positive gain and update its value on the quantization grid to reduce the reconstruction error introduced by re-pruning.
We denote this integration as JSQ+ours. Table E evaluates our two components within JSQ on four widely used LLMs under W4A8KV16 quantization with 2:4 sparsity. JSQ+ours improves average zero-shot accuracy on all four models and reduces WikiText-2 perplexity on three of them. For example, on Qwen3-8B, it reduces perplexity from 40.22 to 35.82, a relative reduction of 10.9%, and increases average zero-shot accuracy from 54.69% to 55.21% (a gain of 0.52 percentage points).
These results demonstrate the effectiveness and extensibility of our proposed method.
Table E: Results for JSQ and JSQ+ours under W4A8KV16 quantization with 2:4 sparsity
| Model | Method | WikiText-2 PPL ↓ | Avg. accuracy (%) ↑ |
|---|---|---|---|
| Llama2-7B | JSQ | 15.96 | 55.40 |
| JSQ+ours | 15.93 | 55.77 | |
| Llama3-8B | JSQ | 134.73 | 42.03 |
| JSQ+ours | 145.03 | 43.64 | |
| Mistral-7B | JSQ | 14.00 | 57.49 |
| JSQ+ours | 13.74 | 58.17 | |
| Qwen3-8B | JSQ | 40.22 | 54.69 |
| JSQ+ours | 35.82 | 55.21 |
Response to Reviewer e9j3 (Part 1/5)
We thank the reviewer for the thoughtful feedback and for recognizing the motivation of our work. Below, we address the reviewer’s concerns point by point.
Thank you for raising this concern. We repeated the comparison on Mistral-7B and Llama2-7B under W8A8KV16 and W4A8KV16 with 2:4 sparsity. The tables below report the mean ± sample standard deviation across five calibration seeds. Average zero-shot accuracy is the arithmetic mean over six tasks.
The results under 8-bit weight quantization are shown in Table A. PQP achieves modest improvements over OBR at this precision. Fewer retained weights are rounded to zero than under 4-bit quantization, leaving less quantization-induced free sparsity for PQP to exploit and less opportunity to reduce explicit pruning.
The results under 4-bit weight quantization are shown in Table B. Our method reduces mean WikiText-2 perplexity by 0.44 on Mistral-7B and 0.59 on Llama2-7B, while increasing average zero-shot accuracy by 0.93 and 1.34 percentage points, respectively. PQP also has lower standard deviations for both metrics on both models in this 4-bit setting. At lower weight bit-widths, quantization produces more zeros, allowing PQP to preserve more weights during the initial pruning stage while satisfying the final 2:4 sparsity constraint. Our main experiments focus on 4-bit weight quantization, where this benefit is more pronounced. These results support the robustness of the gains in our main experimental setting.
Table A: Robustness across calibration seeds under W8A8KV16 quantization with 2:4 sparsity
Table B: Robustness across calibration seeds under W4A8KV16 quantization with 2:4 sparsity