acceptodds
Under review as a conference paper at ICLR 2027

Prune-Quantize-Prune: Towards Better Joint Model Compression by Getting Free Sparsity from Symmetric Quantization

Abstract

The deployment of large language models (LLMs) at scale imposes substantial demands on memory bandwidth and computational throughput, making model compression essential for efficient inference. Among post-training compression techniques, 2:4 semi-structured sparsity and low-bit quantization are particularly attractive, yet effectively combining them remains challenging. Existing approaches typically apply pruning and quantization sequentially, either in a prune-then-quantize or quantize-then-prune manner, without explicitly exploiting the additional sparsity naturally induced by quantization. To address this limitation, we propose a three-stage **prune-quantize-prune** framework tailored to the combination of semi-structured sparsity and symmetric quantization. In the first stage, we introduce a *Quantization-aware Mask Selection* strategy that avoids prematurely enforcing the exact 2:4 constraint and instead reserves room for quantization-induced free sparsity. After applying low-bit symmetric quantization, the resulting additional zeros are explicitly exploited. In the final stage, we re-prune the quantized weights to strictly satisfy the target 2:4 sparsity pattern. To mitigate the reconstruction error introduced by this post-quantization pruning, we further propose a *Grid-constrained Compensation* method, which performs second-order weight compensation directly in the quantized space while preserving the discrete quantization grid. Extensive experimental results on popular LLMs demonstrate that this method outperforms existing state-of-the-art methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Compression, Prune, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_e9j3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Prune-Quantize-Prune (PQP), a joint compression framework that exploits quantization-induced "free sparsity" under symmetric quantization. PQP consists of quantization-aware mask selection, quantization, and grid-constrained compensation to satisfy the target sparsity pattern while minimizing reconstruction error. Experiments across LLaMA and Mistral models demonstrate improvements over existing pruning-quantization baselines.

Critical strengths
  • S1. Interesting motivation. Exploiting quantization-induced free sparsity is an intuitive and interesting approach to joint pruning and quantization
Critical weaknesses
  • W1. Limited empirical improvements. While PQP generally outperforms OBR, the improvements are relatively minor in several settings. For instance, under W8A8KV16 quantization (Table 3), PQP improves average accuracy by only 0.23%p and perplexity by 0.02. Such differences may fall within experimental variability. Moreover, Grid-constrained Compensation yields negligible reconstruction error reductions in several layers (Table 1), raising questions about its practical contribution. Also, proposed PQP is impelmented by adopting Hadamard transformation, and it is not clear whether the method itself is benefiting from rotation, not from the proposed method

  • W2. Insufficient methodological clarity and experimental justification. Several implementation details are unclear, including trial mask construction, the distinction between trial and actual quantization, the role of Hadamard rotation, efficiency evaluations. The experimental evaluation also lacks comparisons with relevant methods such as JSQ, despite discussing them in the background.

Questions for authors

Q1. Given the marginal improvements over OBR in several settings, could the authors provide evidence of statistical significance or robustness across calibration seeds?

Q2. Table 1 shows negligible improvements from Grid-constrained Compensation in several layers. How does this translate to end-to-end perplexity and accuracy improvements?

Q3. In Table 4, does "Quantization-aware mask" exclude Grid-constrained Compensation? Does the final row include both components, or compensation alone?

Q4. Why is JSQ excluded from the experimental comparisons despite being discussed in Section 2.3?

Q5. How does Hadamard rotation affect PQP? Could the authors provide results without rotation?

Q6. Could the authors clarify the kernel backends, matrix dimensions, and workloads used for the speedup evaluation in Section 5.5?

6 comments
AuthorsOpenReview ↗

Response to Reviewer e9j3 (Part 1/5)

We thank the reviewer for the thoughtful feedback and for recognizing the motivation of our work. Below, we address the reviewer’s concerns point by point.

Q1: "Given the marginal improvements over OBR in several settings, could the authors provide evidence of statistical significance or robustness across calibration seeds?"

Thank you for raising this concern. We repeated the comparison on Mistral-7B and Llama2-7B under W8A8KV16 and W4A8KV16 with 2:4 sparsity. The tables below report the mean ± sample standard deviation across five calibration seeds. Average zero-shot accuracy is the arithmetic mean over six tasks.

The results under 8-bit weight quantization are shown in Table A. PQP achieves modest improvements over OBR at this precision. Fewer retained weights are rounded to zero than under 4-bit quantization, leaving less quantization-induced free sparsity for PQP to exploit and less opportunity to reduce explicit pruning.

The results under 4-bit weight quantization are shown in Table B. Our method reduces mean WikiText-2 perplexity by 0.44 on Mistral-7B and 0.59 on Llama2-7B, while increasing average zero-shot accuracy by 0.93 and 1.34 percentage points, respectively. PQP also has lower standard deviations for both metrics on both models in this 4-bit setting. At lower weight bit-widths, quantization produces more zeros, allowing PQP to preserve more weights during the initial pruning stage while satisfying the final 2:4 sparsity constraint. Our main experiments focus on 4-bit weight quantization, where this benefit is more pronounced. These results support the robustness of the gains in our main experimental setting.

Table A: Robustness across calibration seeds under W8A8KV16 quantization with 2:4 sparsity

ModelMethodWikiText-2 PPL ↓Avg. accuracy (%) ↑
Mistral-7BSparseGPT+GPTQ8.89 ± 0.3058.54 ± 0.82
OBR8.15 ± 0.1359.96 ± 0.60
PQP (ours)8.13 ± 0.0860.12 ± 0.50
Llama2-7BSparseGPT+GPTQ16.28 ± 1.3546.81 ± 0.66
OBR9.02 ± 0.1156.70 ± 0.50
PQP (ours)8.98 ± 0.1156.73 ± 0.45

Table B: Robustness across calibration seeds under W4A8KV16 quantization with 2:4 sparsity

ModelMethodWikiText-2 PPL ↓Avg. accuracy (%) ↑
Mistral-7BSparseGPT+GPTQ9.88 ± 0.1656.65 ± 0.92
OBR8.96 ± 0.0758.06 ± 0.52
PQP (ours)8.52 ± 0.0458.99 ± 0.30
Llama2-7BSparseGPT+GPTQ18.70 ± 1.7946.11 ± 0.79
OBR10.46 ± 0.2753.43 ± 0.57
PQP (ours)9.87 ± 0.1354.77 ± 0.22
AuthorsOpenReview ↗

Response to Reviewer e9j3 (Part 2/5)

Q2: "Table 1 shows negligible improvements from Grid-constrained Compensation in several layers. How does this translate to end-to-end perplexity and accuracy improvements?"

Thank you for raising this question. We analyze calibration reconstruction error for all seven linear projections (Q, K, V, O, Gate, Up, and Down) in every Transformer layer: 224 projections in Llama2-7B and 280 in Llama2-13B. We compare our Grid-constrained Compensation with direct zeroing under W4A16 quantization and 2:4 sparsity. For each projection pp, the relative reconstruction error reduction is (Ep,direct−Ep,gc)/Ep,direct(E_{p,\mathrm{direct}}-E_{p,\mathrm{gc}})/E_{p,\mathrm{direct}}, where Ep,directE_{p,\mathrm{direct}} and Ep,gcE_{p,\mathrm{gc}} denote the reconstruction errors under direct zeroing and Grid-constrained Compensation, respectively. As shown in Table C, Grid-constrained Compensation reduces calibration reconstruction error in every measured projection, with a maximum reduction of about 75%. Lower layer-wise reconstruction error across these projections better preserves projection outputs and is expected to benefit downstream tasks.

Table D reports end-to-end ablations on both Llama2-7B and Llama2-13B. On Llama2-7B, adding Grid-constrained Compensation to Quantization-aware Mask Selection reduces WikiText-2 perplexity from 9.84 to 9.77 and improves average zero-shot accuracy from 54.40% to 54.69%. On Llama2-13B, the same addition reduces perplexity from 8.05 to 7.84 and improves average zero-shot accuracy from 59.94% to 60.47%. These results show measurable end-to-end improvements on both models alongside the reconstruction gains.

Table C: Calibration reconstruction error reduction with Grid-constrained Compensation relative to direct zeroing

ModelImproved projections ↑Max. reduction (%) ↑Min. reduction (%) ↑Median reduction (%) ↑
Llama2-7B224/22475.10510.01996.4382
Llama2-13B280/28071.85930.01115.6959

Table D: End-to-end ablation of PQP components on Llama2-7B and Llama2-13B under W4A16KV16 quantization with 2:4 sparsity

ModelConfigurationWikiText-2 PPL ↓Avg. accuracy (%) ↑
Llama2-7BOBR10.3854.12
PQP w/o Grid-constrained Compensation9.8454.40
PQP w/ Grid-constrained Compensation9.7754.69
Llama2-13BOBR8.2959.59
PQP w/o Grid-constrained Compensation8.0559.94
PQP w/ Grid-constrained Compensation7.8460.47

Q3: "In Table 4, does “Quantization-aware mask” exclude Grid-constrained Compensation? Does the final row include both components, or compensation alone?"

Thank you for requesting this clarification. We explain below how the two rows in Table 4 of the manuscript differ and which components are included in each configuration:

  1. The “Quantization-aware mask” row includes Quantization-aware Mask Selection but excludes Grid-constrained Compensation. This configuration isolates the contribution of Quantization-aware Mask Selection relative to the OBR baseline. Specifically, after quantization, weights selected for additional pruning to satisfy the 2:4 sparsity constraint are directly set to zero, without any compensation or adjustment to the remaining weights at this stage.
  2. The final row includes both Quantization-aware Mask Selection and Grid-constrained Compensation. Comparing this row with the “Quantization-aware mask” row isolates the additional contribution of Grid-constrained Compensation. It uses the same mask-selection procedure and additionally compensates for the reconstruction error introduced by final re-pruning by updating the retained weights while keeping them on the quantization grid.

We have revised the manuscript text and Table 4 to make this cumulative relationship explicit.

AuthorsOpenReview ↗

Response to Reviewer e9j3 (Part 3/5)

Q4: "Why is JSQ excluded from the experimental comparisons despite being discussed in Section 2.3?"

Thank you for pointing out this missing comparison. Our method was developed primarily based on OBR's compensation framework, so we initially focused on OBR as the main baseline and did not include JSQ in the experimental comparisons. We agree that JSQ is a relevant baseline. To evaluate whether our proposed components are also effective within the JSQ pipeline, we have now added a comparison between JSQ and JSQ augmented with these components (Table E).

Our proposed components can also be integrated into JSQ. Specifically, JSQ introduces a sparsity metric that accounts for both pruning importance and quantization sensitivity, together with an activation editor to mitigate outliers (Guo et al., 2024). We integrate our Quantization-aware Mask Selection and Grid-constrained Compensation into this pipeline as follows:

  1. Quantization-aware Mask Selection. We first construct a standard 2:4 mask using JSQ's importance score and perform trial smoothing and RTN quantization. Within each group of four weights, we count the retained weights that are quantized to zero and use this count as the budget for restoring initially pruned weights, prioritizing those with higher JSQ scores. We then apply the relaxed mask to the original weights and repeat smoothing and quantization, allowing quantization-induced zeros to contribute to the target sparsity.
  2. Grid-constrained Compensation. After actual quantization, we re-prune groups with more than two nonzero weights. For each weight removed during final re-pruning, we use our proposed method to compute the compensation gain for each candidate retained weight. We then select the candidate with the largest positive gain and update its value on the quantization grid to reduce the reconstruction error introduced by re-pruning.

We denote this integration as JSQ+ours. Table E evaluates our two components within JSQ on four widely used LLMs under W4A8KV16 quantization with 2:4 sparsity. JSQ+ours improves average zero-shot accuracy on all four models and reduces WikiText-2 perplexity on three of them. For example, on Qwen3-8B, it reduces perplexity from 40.22 to 35.82, a relative reduction of 10.9%, and increases average zero-shot accuracy from 54.69% to 55.21% (a gain of 0.52 percentage points).

These results demonstrate the effectiveness and extensibility of our proposed method.

Table E: Results for JSQ and JSQ+ours under W4A8KV16 quantization with 2:4 sparsity

ModelMethodWikiText-2 PPL ↓Avg. accuracy (%) ↑
Llama2-7BJSQ15.9655.40
JSQ+ours15.9355.77
Llama3-8BJSQ134.7342.03
JSQ+ours145.0343.64
Mistral-7BJSQ14.0057.49
JSQ+ours13.7458.17
Qwen3-8BJSQ40.2254.69
JSQ+ours35.8255.21
Reviewer_pzvROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a prune–quantize–prune framework that anticipates quantization-induced zeros to adjust initial pruning masks. A final pruning stage enforces structured sparsity, with grid-constrained compensation to preserve quantization levels. Experiments evaluate reconstruction error, perplexity, commonsense accuracy, calibration costs, and hardware latency.

Critical strengths

The method addresses a meaningful interaction between pruning and quantization: weights retained during pruning may subsequently become zero after quantization. The reported improvements across several models and component ablations provide initial evidence that accounting for this interaction can improve accuracy under fixed sparsity and precision constraints.

Critical weaknesses
  1. The necessity and reliability of the three-stage procedure are insufficiently established. A quantize–then–prune baseline using the same grid-constrained compensation is missing, leaving the independent benefit of trial-based mask prediction unclear. Most reported accuracy improvements over OBR are modest, without repeated-run uncertainty estimates, despite approximately doubling calibration time.
  2. Capability preservation and practical deployment benefits remain insufficiently validated. Evaluation is limited to perplexity and commonsense tasks, leaving autoregressive reasoning and instruction-following capabilities untested. The hardware benchmark lacks sufficient workload and implementation details to establish end-to-end benefits for the evaluated compressed models.
Questions for authors
  1. With the final sparsity, quantizer, and compensation strategy held fixed, how does PQP compare with quantize–then–prune? What evidence isolates the benefit of the additional trial-based prediction stage?
  2. Do the gains persist on generative benchmarks for instruction-tuned models? What exact model or operator, kernel, and measurement procedure produced the latency results in Figure 3?
2 comments
AuthorsOpenReview ↗

Response to Reviewer pzvR (Part 1/2)

We thank the reviewer for the thoughtful feedback. Below, we address the reviewer’s concerns point by point.

Q1: "With the final sparsity, quantizer, and compensation strategy held fixed, how does PQP compare with quantize–then–prune? What evidence isolates the benefit of the additional trial-based prediction stage?"

Thank you for these questions. We conducted the following experiments.

Compare with quantize–then–prune. We compare the compression pipelines on Llama2-7B under W4A16KV16 and the 2:4 structural constraint, with RTN quantization. The Quant-Prune baseline applies a Wanda 2:4 mask to RTN-quantized weights, directly zeroing the selected positions. The OBR and ours rows reuse the Standard 2:4 and our quantization-aware results from the controlled RTN experiment in Table B, respectively.

Table A: Compression pipeline comparison on Llama2-7B using RTN quantization (W4A16KV16, 2:4 sparsity)

MethodWikiText-2 PPL ↓Avg. zero-shot accuracy (%) ↑
Quant-Prune18.4547.94
Prune-Quant (OBR)11.7450.88
Prune-Quant-Prune (ours)11.0152.33

Quant-Prune already accounts for quantization-induced zeros when selecting its mask. However, once weights are quantized, the continuous optimal correction for pruning error may lie outside the quantization grid. PQP uses trial predictions to anticipate these zeros and guide initial pruning, allowing compensation in continuous weight space before actual quantization. It then applies final re-pruning with grid-constrained compensation to enforce the 2:4 constraint.

Benefit of the additional trial-based prediction stage. Table B isolates the contribution of trial prediction on Llama2-7B at W4A16KV16. All three variants use the same RTN quantizer and frozen quantization grid. We compare three variants:

  • Standard 2:4: use the standard Wanda 2:4 mask without prediction-driven relaxation.
  • Trial-based prediction (random): execute the same trial pass as full PQP, but randomly permute the predicted extra-zero counts among four-weight groups within each output channel. This preserves the trial weights and scores, the numbers of groups retaining two, three, or four weights, and the total pre-pruning budget in each output channel, while breaking the correspondence between each prediction and its original group.
  • Trial-based prediction (our quantization-aware): use the trial predictions for their corresponding groups to determine adaptive pre-pruning.

Table B shows that our quantization-aware prediction lowers WikiText-2 perplexity from 11.74 to 11.01 and improves accuracy from 50.88% to 52.33% relative to Standard 2:4. Randomly permuting predictions yields 14.90 PPL and 49.45% accuracy, despite preserving the trial computation and pre-pruning budget. This supports the value of assigning predictions to the correct groups.

Table B: Mask-selection ablation on Llama2-7B using RTN quantization (W4A16KV16, 2:4 sparsity)

Mask selectionWikiText-2 PPL ↓Avg. zero-shot accuracy (%) ↑
Standard 2:411.7450.88
Trial-based prediction (random)14.9049.45
Trial-based prediction (our quantization-aware)11.0152.33
AuthorsOpenReview ↗

Response to Reviewer pzvR (Part 2/2)

Q2: "Do the gains persist on generative benchmarks for instruction-tuned models? What exact model or operator, kernel, and measurement procedure produced the latency results in Figure 3?"

Generative evaluation. We evaluate Qwen3-8B on GSM8K, IFEval, and HumanEval. Following the MLP-only pruning scope in the semi-structured experiments of RAC, we apply joint 2:4 sparsity and 4-bit quantization only to the MLP projections. We adapt the calibration data following RAC by including task prompts and responses generated by the corresponding dense model.

Table C: Generative evaluation on Qwen3-8B with MLP-only 2:4 sparsity and 4-bit quantization

MethodGSM8K (%) ↑IFEval (%) ↑HumanEval pass@1 (%) ↑
OBR52.6955.0851.22
PQP (ours)54.5160.4457.32

In Table C, PQP improves over OBR by 1.82, 5.36, and 6.10 percentage points on GSM8K, IFEval, and HumanEval, respectively. These results extend the evidence beyond perplexity and commonsense tasks, supporting PQP's effectiveness on generative benchmarks under the tested MLP-only compression setting.

Latency measurements in Figure 3. The figure reports a synthetic linear-projection microbenchmark on a single NVIDIA A100-SXM4-80GB GPU. The input and output dimensions are both 4096. With batch size BB, sequence length SS, and total token count T=BST=BS, the operator is Y=XW⊤Y=XW^\top, where X∈RT×4096X\in\mathbb{R}^{T\times4096} and W∈R4096×4096W\in\mathbb{R}^{4096\times4096}. Figure 3 presents equivalent workloads at S=512S=512 and B∈{16,32,64,128,256}B\in\{16,32,64,128,256\}, corresponding to CUTLASS GEMM dimensions (M,N,K)=(4096,512B,4096)(M,N,K)=(4096,512B,4096).

The INT4 dense and INT4 2:4 sparse implementations use CUTLASS Gemm and SparseGemm, respectively, targeting SM80 Tensor Cores, with INT4 weights and activations and INT32 accumulation/output. The FP16 dense baseline uses PyTorch CUDA torch.matmul. We use three warm-up calls for the CUTLASS operators and five for the FP16 baseline, then report the mean latency of five timed calls. Each timed call ends with CUDA synchronization; wall-clock timing includes operator setup, kernel launch, and synchronization overhead. Random input generation, sparse metadata preparation, and host-to-device transfers occur before timing.

W1: "The necessity and reliability of the three-stage procedure are insufficiently established. A quantize–then–prune baseline using the same grid-constrained compensation is missing, leaving the independent benefit of trial-based mask prediction unclear. Most reported accuracy improvements over OBR are modest, without repeated-run uncertainty estimates, despite approximately doubling calibration time."

Thank you for raising these concerns. Our response to Q1 adds a Quant-Prune baseline and a separate controlled ablation of trial prediction. To assess robustness to calibration data, we also repeated the comparison on Mistral-7B and Llama2-7B under W4A8KV16 with 2:4 sparsity. Table D reports the mean ± sample standard deviation across five calibration seeds. Average zero-shot accuracy is the arithmetic mean over six tasks.

Our method reduces mean WikiText-2 perplexity by 0.44 on Mistral-7B and 0.59 on Llama2-7B, while increasing average zero-shot accuracy by 0.93 and 1.34 percentage points, respectively. PQP also has lower standard deviations for both metrics on both models. These results support the robustness of the gains in our main 4-bit weight quantization setting.

Table D: Robustness across calibration seeds under W4A8KV16 quantization with 2:4 sparsity

ModelMethodWikiText-2 PPL ↓Avg. accuracy (%) ↑
Mistral-7BSparseGPT+GPTQ9.88 ± 0.1656.65 ± 0.92
OBR8.96 ± 0.0758.06 ± 0.52
PQP (ours)8.52 ± 0.0458.99 ± 0.30
Llama2-7BSparseGPT+GPTQ18.70 ± 1.7946.11 ± 0.79
OBR10.46 ± 0.2753.43 ± 0.57
PQP (ours)9.87 ± 0.1354.77 ± 0.22

W2: "Capability preservation and practical deployment benefits remain insufficiently validated. Evaluation is limited to perplexity and commonsense tasks, leaving autoregressive reasoning and instruction-following capabilities untested. The hardware benchmark lacks sufficient workload and implementation details to establish end-to-end benefits for the evaluated compressed models."

These concerns are addressed in our response to Q2 above, where we provide additional generative evaluations and clarify the hardware benchmark’s workload, implementation, measurement protocol, and operator-level scope.


We hope these clarifications address your concerns.

Reviewer_TAWgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies post-training compression of LLMs by combining 2:4 semi-structured sparsity with symmetric low-bit quantization. PQP estimates quantization-induced zeros through a trial pass, adjusts the initial pruning quota, and then re-prunes quantized weights with a Hessian-based compensation rule that updates one retained weight. Evaluation covers several LLMs using WikiText2 perplexity and 6 zero-shot tasks, supplemented by component ablations, sparsity analyses, alternative compression configurations, and runtime measurements.

Critical strengths
  1. PQP offers a specific extension beyond OBR, which already observes quantization-induced sparsity and compensates weights under a given pruning mask before final quantization. Its methodological contribution is to use trial zero counts to modify each group’s initial pruning quota, directly connecting expected quantization behavior to mask construction.
  2. The reported improvements provide empirical value across model sizes and support contributions from both proposed components. PQP improves perplexity and mean zero-shot accuracy over OBR on all five models in Tab. 2. The LLaMA2-13B ablation reduces perplexity from 8.29 to 8.05 to 7.84 as mask adaptation and compensation are added, while experiments with RTN and GPTQ support applicability across these two quantizers.
Critical weaknesses
  1. The method is insufficient to verify that the evaluated implementation realizes the proposed constrained compression pipeline. Grid alignment alone does not establish finite-bit representability, and the pseudocode does not consistently implement the adaptive mask described in the main text.
  2. The identical LLaMA2-7B baseline results in Tabs. 2 and 8 require clarification because the tables specify different activation precisions. These inconsistencies leave the actual baseline configuration and the comparability of the reported results unclear.
Questions for authors
  1. How does grid-constrained compensation enforce the finite codebook bounds defined in Eq. 4, beyond the lattice rounding in Eq. 12? Please clarify whether the reported results include any clipping or candidate rejection required to preserve the claimed bit-width.
  2. Can you provide implementation-consistent pseudocode for the first two stages, explaining why Alg. 1 line 16 uses M0M_0 rather than the newly constructed MqM_q, and where the trial pruning compensation described in Sec. 4.1 is executed?
  3. Please clarify whether SparseGPT+GPTQ was implemented as sequential pruning followed by quantization or as a joint pruning and quantization procedure. Section 5.1 suggests the former, whereas Appendix C describes the latter when discussing the 2-bit results.
  4. Please explain why the SparseGPT+GPTQ results for LLaMA2-7B under 2:4 sparsity are identical in Tables 2 and 8, including perplexity 21.08 and mean accuracy 46.01%, despite the different stated quantization settings.
  5. Please fix the minor presentation errors in Sec. 5.3 by changing the Mistral reference from Tab. 8 to Tab. 3 and describing Tab. 2 as covering five models. The perplexity change from 8.97 to 8.51 and accuracy change from 58.47% to 59.30% belong to W4A8KV16, whereas the W4A4KV4 results are 10.91 -> 10.32 and 53.03% -> 53.48%. Please replace “PTP” with “PQP”, change “third pruning phase” to “second pruning phase” or “third stage”, and remove the repeated sentence. Please also consolidate the duplicate Tseng 2024a/2024b entries for QuIP#, add the missing QuIP reference, and correct “rece,” “accordingg,” “accurancy,” “Actural,” and “Ruantization”.
3 comments
AuthorsOpenReview ↗

Response to Reviewer TAWg (Part 1/3)

We thank the reviewer for the thoughtful feedback and for recognizing the method and results. Below, we address the reviewer’s concerns point by point.


Q1: "How does grid-constrained compensation enforce the finite codebook bounds defined in Eq. 4, beyond the lattice rounding in Eq. 12? Please clarify whether the reported results include any clipping or candidate rejection required to preserve the claimed bit-width."

Thank you for pointing out this omission. In the original manuscript, Eq. 12 presented only the lattice-rounding operation and omitted the finite-codebook clipping step used in our implementation.

For each weight wqw_q to be pruned, we consider only nonzero retained weights wjw_j in the same row as compensation candidates, excluding all positions selected for pruning. Eq. 11 gives the theoretical continuous compensation cjcontc_j^{\mathrm{cont}}. Under the quantization grid and finite-codebook constraints, the actual compensation cj⋆c_j^\star must keep the updated weight on the grid and and within the representable range of the finite codebook. It satisfies srqmin⁡≤wj+cj⋆≤srqmax⁡s_rq_{\min}\le w_j+c_j^\star\le s_rq_{\max}, where srs_r is the quantization step size and qmin⁡,qmax⁡q_{\min},q_{\max} are the integer codebook bounds defined in Eq. 4. We have therefore revised Eq. 12 to explicitly include clipping:

s_r\operatorname{round}\left(\frac{c_j^{\mathrm{cont}}}{s_r}\right), s_rq_{\min}-w_j,\;s_rq_{\max}-w_j \right).$$ The compensation residual $\varepsilon_j=c_j^\star-c_j^{\mathrm{cont}}$ is the difference between the actual compensation in Eq. 12 and the theoretical compensation in Eq. 11, caused by the grid and codebook constraints. We evaluate the gain using the actual compensation after clipping and select the candidate with the largest positive gain. **By restricting the compensated weights to the original quantization grid and codebook, our method preserves the target bit-width without introducing additional quantization levels**. We have clarified the clipping operation, the compensation residual, and the nonzero candidate restriction in Sec. 4.3 and Algorithm 1, and highlighted the revisions in red in the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer TAWg (Part 2/3)


Q2: "Can you provide implementation-consistent pseudocode for the first two stages, explaining why Alg. 1 line 16 uses M0M_0 rather than the newly constructed MqM_q, and where the trial pruning compensation described in Sec. 4.1 is executed?"

We thank the reviewer for pointing out these issues. The use of M0M_0 instead of MqM_q in line 16 of the original Algorithm 1 was a notation error; that line should quantize the weights under the newly constructed adaptive mask MqM_q. The trial pruning compensation described in Sec. 4.1 is executed after trial pruning and before trial quantization. We have corrected Algorithm 1 in the revised manuscript. The pseudocode below makes the trial and actual passes explicit:

Algorithm 1. PQP - Stages I and II

Input: Dense weights W, importance scores S, Hessian H, target sparsity N:M (N zeros per group), and b-bit quantizer Q_b.

 1  Stage I: Pruning-Quantization-Aware Mask
 2  for each N:M group g of M weights do                 // Trial pruning
 3      M0[j] = 1 at the M-N largest S[j] in g; otherwise 0
 4  end for
 5  W_comp = W * M0                                     // Element-wise product
 6  for each output row r do
 7      I = {j : M0[r,j] = 1}; Z = {j : M0[r,j] = 0}
 8      W_comp[r,I] = W[r,I] + transpose(delta_r)        // Optimal second-order error compensation
 9  end for
10  W_trial = Q_b(W_comp)                               // Trial quantization
11  Z_q = (M0 == 1) * (W_trial == 0)                     // Element-wise: retained, then quantized to zero
12  for each N:M group g of M weights do                // Actual pruning
13      z_g = sum(Z_q[j] for j in g)                    // Estimated quantization-induced zeros in g
14      n_g = max(0, N - z_g)                           // Number to prune in g
15      K_g = positions of the M-n_g largest S[j] in g  // Positions to retain
16      M_q[j] = 1 if j is in K_g; otherwise 0, for j in g
17  end for
18  U_q = W * M_q                                      // Element-wise product
19  for each output row r do
20      I = {j : M_q[r,j] = 1}; Z = {j : M_q[r,j] = 0}
21      U_q[r,I] = W[r,I] + transpose(delta_r)           // Optimal second-order error compensation
22  end for
23  Stage II: Quantization
24  W_bar_q = Q_b(U_q)                                  // Actual quantization
25  return W_bar_q, M_q

Our first and second stages involve the following steps:

  • Trial pruning. The trial mask is a standard 2:4 sparsity mask that prunes two weights in each group of four. Trial pruning compensation is performed at this stage.
  • Trial quantization. After trial pruning and compensation, trial quantization is applied to the weights under the standard 2:4 mask. Its output only guides the construction of the actual pruning mask; it is not the actual quantization result.
  • Actual pruning. A relaxed pruning mask is constructed based on the estimated quantization-induced zeros, allowing fewer than two weights to be pruned in some groups. The relaxed mask is applied to the original dense weights, and the pruning-error compensation is recomputed.
  • Actual quantization. Actual quantization is performed on the compensated weights. The resulting quantized weights are passed to the second pruning stage, which enforces the 2:4 constraint.

We have revised Algorithm 1 to clearly distinguish the trial and actual pruning–quantization passes and explicitly indicate where pruning-error compensation is performed. The corresponding changes are highlighted in red in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer TAWg (Part 3/3)


Q3: "Please clarify whether SparseGPT+GPTQ was implemented as sequential pruning followed by quantization or as a joint pruning and quantization procedure. Section 5.1 suggests the former, whereas Appendix C describes the latter when discussing the 2-bit results."

Thank you for pointing out this ambiguity. Our SparseGPT+GPTQ baseline follows the official code released with OBR. It is a joint pruning and quantization procedure in terms of its error compensation. SparseGPT and GPTQ use the same column-wise, second-order compensation framework: after processing a weight column, they update the remaining columns to compensate for the error introduced by that operation. SparseGPT compensates for pruning error, while GPTQ compensates for quantization error. In this implementation, both pruning and quantization errors are accounted for during the column-wise updates, so the compensation reflects their combined effect rather than applying quantization only after an independently completed pruning pass. We have clarified the SparseGPT+GPTQ baseline description in Sec. 5.1 of the revised manuscript and highlighted the revision in red.


Q4: Please explain why the SparseGPT+GPTQ results for LLaMA2-7B under 2:4 sparsity are identical in Tables 2 and 8, including perplexity 21.08 and mean accuracy 46.01%, despite the different stated quantization settings.

Thank you for pointing out this inconsistency. We checked the experiment logs and corrected the affected entries in Table 8 and highlighted them in red in the revised manuscript.

Q5: Please fix the minor presentation errors in Sec. 5.3 by changing the Mistral reference from Tab. 8 to Tab. 3 and describing Tab. 2 as covering five models. The perplexity change from 8.97 to 8.51 and accuracy change from 58.47% to 59.30% belong to W4A8KV16, whereas the W4A4KV4 results are 10.91 -> 10.32 and 53.03% -> 53.48%. Please replace “PTP” with “PQP”, change “third pruning phase” to “second pruning phase” or “third stage”, and remove the repeated sentence. Please also consolidate the duplicate Tseng 2024a/2024b entries for QuIP#, add the missing QuIP reference, and correct “rece,” “accordingg,” “accurancy,” “Actural,” and “Ruantization”.

We sincerely thank the reviewer for reading our paper so carefully and identifying these errors. We have corrected the citations and other minor presentation errors in the revised version.

We hope these clarifications address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.