- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GapSQ to improve language model compression pipelines that apply pruning followed by low-bit quantization. The authors observe that pruning leaves a gap between the retained positive and negative weights, yet standard uniform quantization allocates part of its representable range to this region. GapSQ shifts the two branches inward within each group and uses a strict separability condition on the quantization codes, together with a group-level reference code, to determine the inverse-shift direction during dequantization. The shift magnitude is selected based on local output error evaluated using calibration activations. The paper reports accuracy improvements across multiple models, pruning methods, and quantization methods, and provides comparisons with matched metadata budgets, low-bit sparse kernels, and end-to-end latency evaluations.
Critical strengths
- The paper offers a simple, implementable solution to a specific distributional issue in joint compression. Strict separability in the quantization-code domain and a group-level reference code allow the shifts applied to the two branches to be reversed during dequantization without introducing per-weight sign bits. The method preserves the existing sparsity structure and can be fused into low-bit sparse kernels.
- The experimental evidence covers a broad range of settings, and key controls help rule out alternative explanations. The main results show improvements across all four models and both bit widths. The appendices cover additional pruning and quantization backbones, different calibration datasets, comparisons with matched metadata budgets, and measured additional overhead relative to Sparse-Marlin. In particular, Table 16 shows that the gains are not merely attributable to an increased metadata budget.
Critical weaknesses
1.The main text’s claim of the “same storage budget” as dense 3-bit quantization remains insufficiently precise. The appendix itself reports that GapSQ requires 3.3125 bits per original weight with 4-bit quantization, 50% sparsity, and a group size of 128. Different formats also incur their own storage costs for scales, zero-points, masks, and actual packing overhead. Table 16 addresses metadata fairness among sparse methods, but does not fully substantiate the main text’s claim of a matched storage budget relative to the dense baselines. I would like to see actual model sizes reported in bytes, together with comparisons under matched actual storage budgets. 2.The connection between the key recoverability condition and the actual quantization pipeline is not sufficiently detailed. Section 4 checks branch separability using candidate quantization codes, but does not adequately explain whether this condition is still guaranteed after GPTQ error compensation and storage of the final parameters, or how groups with retained weights of only one sign or no retained weights are handled. Results under aggressive compression may be sensitive to the underlying configuration; for example, the 3-bit baseline PPL for Llama 3-3B is 359.47. Complete pseudocode, detailed quantization settings, and results from an appropriate number of repeated calibration runs would strengthen reproducibility.
Questions for authors
- Please report the actual model storage footprint, including masks, all metadata, and packing overhead, and revisit the claim of the “same storage budget” relative to the dense 3-bit baseline. Do the main conclusions regarding accuracy still hold when the actual storage sizes in bytes are strictly matched? The existing metadata-matched comparisons between sparse methods in Table 16 are helpful.
- At exactly which stage is the branch separability condition checked? Is this condition still guaranteed after GPTQ error propagation, FP16 storage of the shift magnitude, and final packing of the quantization codes? How are groups containing retained weights of only one sign, groups with no retained weights, and the zero-shift fallback handled? Please provide pseudocode corresponding to the main experiments.
- Please provide the damping, clipping, grouping, and calibration settings used for the severely degraded 3-bit baselines, and report results using a few differently sampled calibration sets. Do GapSQ’s substantial improvements primarily arise from robustly exploiting the gap, or do they depend on specific baseline configurations? These clarifications would affect my assessment of the strength of the reported gains.
6 comments
Authors' Response to Reviewer FwBM (2/3)
2. Details of branch separability check (Question 2 and the first half of Weakness 2)
-
Integration with GPTQ
For generality, separability is checked on the final group codes, after all within-group GPTQ error propagation. If these updates cause a weight to be quantized outside its original branch’s allowed code range, we discard that shift candidate, with the base quantizer with retained as a fallback, as described in and explained below. On Llama 3-8B at 50% sparsity with 4-bit GPTQ quantization, only 4.46% of groups finally use this fallback.
After finalizing each group, its error is propagated only to later, unquantized groups. Thus, strict branch separability is preserved throughout compression for every group using a nonzero shift.
-
Special cases and zero-shift fallback
Neither groups with no retained weights nor groups containing only one sign actually occurred in our main experiments. If encountered, both cases use the base quantizer with ; empty groups reconstruct as zeros under the pruning mask.
When , inverse shifting leaves the base quantizer’s output unchanged, enabling safe fallback. From ,
=D_{\mathrm{GPTQ}}(q_i).$$ The fallback therefore uses standard GPTQ codes and parameters with $s=0$. Here, any reference level is valid, and branch separation is unnecessary. -
Pseudocode for details
For clarity, we provide the following procedure.
GPTQ_groupincludes within-group error propagation while keeping masked positions zero. Each candidate starts from the same saved group weights; later groups are updated only after the best candidate is selected. Here,*denotes elementwise multiplication.for each Linear: for each group with weights W_g and pruning mask M_g: W = M_g * copy(W_g) R_pos, R_neg = positive/negative retained indices of W best = GPTQ_group(copy(W), M_g) best.shift, best.reference = 0, 0 best.error = output_MSE(W, M_g * decode(best), X_g) if the group has retained weights and both signs are present: for each nonzero shift candidate s: U = M_g * (W - s * sign(W)) candidate = GPTQ_group(copy(U), M_g) Q = candidate's final codes r = min(Q[R_pos]) if max(Q[R_neg]) >= r: continue W_hat = M_g * inverse_shift(decode(candidate), s, r) error = output_MSE(W, W_hat, X_g) if error < best.error: save candidate with shift s, reference r, and error as best commit best propagate its group error to later, unquantized groups following the standard GPTQ procedureWe will clarify these details in the revision.
Authors' Response to Reviewer FwBM (3/3)
3. Quantization settings and calibration robustness (Question 3 and the second half of Weakness 2)
-
Detailed GPTQ configuration
For the Llama 3-3B baseline results highlighted by the reviewer (SparseGPT+GPTQ, 50% sparsity, 3-bit quantization, PPL 359.47), we used the damping, clipping-search, and block-size settings from the official GPTQ implementation for OPT. The configuration is listed below. Group size and calibration settings are already reported in .
Setting Value Damping 0.01 Clipping search Disabled Quantization group size 128 GPTQ block size 128 Calibration dataset C4 Calibration samples 128 Tokens per sample 2048 -
Results with different calibration seeds and datasets
We repeated the experiment with two additional C4 sampling seeds and also tested WikiText-2 calibration. Each row uses the same calibration data for both methods. All results below use Llama 3-3B at 50% sparsity with 3-bit quantization.
Calibration setting SparseGPT+GPTQ PPL ↓ GapSQ PPL ↓ SparseGPT+GPTQ avg. accuracy ↑ GapSQ avg. accuracy ↑ C4, original run in Tab. 2 359.47 19.48 47.6% 57.8% C4, additional sampling seed A 320.60 19.27 47.9% 57.4% C4, additional sampling seed B 336.72 19.33 47.4% 57.9% WikiText-2 42.56 16.86 49.5% 56.2% GapSQ improves both metrics in every tested setting. Thus, the gains are not limited to one particular sampling seed or to C4. For the reviewer’s reference, we also report GapSQ’s performance with C4, WikiText-2, and Pile calibration on Llama 3-8B in .
-
Why are the 3-bit PPL gains particularly large?
One favorable explanation is that low-bit quantization introduces larger errors, making gap handling more beneficial. With only eight levels at 3 bits, narrowing the pruning-induced gap reduces the represented range and allows finer quantization, which may help alleviate a quantization bottleneck under aggressive compression. This is consistent with , where GapSQ achieves larger absolute PPL reductions at 3 bits than at 4 bits across all four models.
We hope this discussion clarifies the design and effectiveness of our method, resolves your main concerns, and encourages a positive reassessment of our work. We remain happy to answer any follow-up questions.
Authors' Response to Reviewer FwBM (1/3)
We thank the reviewer for the time and effort spent evaluating our paper. Below, we address each concern in detail.
1. Storage-matched comparison (Question 1 and Weakness 1)
GapSQ remains effective under strictly aligned storage accounting, including weights, masks, and metadata. For the reviewer’s reference, we provide comparisons against dense and sparse baselines below, using group size 256 for GapSQ to align the total storage accounting.All results use Llama 3-8B. GPTQ baselines are from Tab. 2; OmniQuant and AWQ baselines are from Appendix C, Tab. 11. * denotes new experiments. Storage entries are in bits per original weight (bpw).
GapSQ still outperforms the corresponding dense and sparse baselines under the same storage accounting.
We have also corrected the imprecise “same storage budget” wording in Line 312. Our intention was to match the weight-plus-mask bit-width while keeping the same quantization group size. Therefore, we also described shift-related metadata as additional group-level overhead in Sec. 4.4 for clarity, with detailed accounting in Appendix H.