acceptodds
Under review as a conference paper at ICLR 2027

GapSQ: Closing the Pruning-Induced Gap for Joint Sparsification and Quantization of LLMs

Abstract

Post-training sparsification and quantization can be combined to compress large language models, but pruning changes the weight distribution seen by the quantizer in a systematic way. By removing many near-zero weights, it creates a pruning-induced gap between the retained negative and positive branches, leading to under-utilization of the quantization range and available levels. Motivated by this observation, we introduce GapSQ, a training-free method for aggressive joint compression. We primarily study 50% sparsity with 3- or 4-bit quantization, where GapSQ narrows the pruning-induced gap through group-wise inward shifting before quantization. To avoid storing per-weight branch information, GapSQ enforces quantized-space separability, allowing the shift to be reversed using only lightweight group-level metadata. Across Llama and Qwen models, multiple sparsity patterns, and several pruning–quantization backbones, GapSQ consistently improves joint-compression baselines, with larger gains observed in more aggressive low-bit regimes. For example, on Llama 3-8B with 3-bit quantization and 50% sparsity, GapSQ improves average zero-shot accuracy from 57.2% to 62.5% while reducing WikiText-2 perplexity from 30.52 to 11.72 relative to SparseGPT+GPTQ. These results show that explicitly adapting post-pruning weight distributions is an effective way to improve low-bit joint compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, model compression, post-training, training-free, pruning, quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FwBMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GapSQ to improve language model compression pipelines that apply pruning followed by low-bit quantization. The authors observe that pruning leaves a gap between the retained positive and negative weights, yet standard uniform quantization allocates part of its representable range to this region. GapSQ shifts the two branches inward within each group and uses a strict separability condition on the quantization codes, together with a group-level reference code, to determine the inverse-shift direction during dequantization. The shift magnitude is selected based on local output error evaluated using calibration activations. The paper reports accuracy improvements across multiple models, pruning methods, and quantization methods, and provides comparisons with matched metadata budgets, low-bit sparse kernels, and end-to-end latency evaluations.

Critical strengths
  1. The paper offers a simple, implementable solution to a specific distributional issue in joint compression. Strict separability in the quantization-code domain and a group-level reference code allow the shifts applied to the two branches to be reversed during dequantization without introducing per-weight sign bits. The method preserves the existing sparsity structure and can be fused into low-bit sparse kernels.
  2. The experimental evidence covers a broad range of settings, and key controls help rule out alternative explanations. The main results show improvements across all four models and both bit widths. The appendices cover additional pruning and quantization backbones, different calibration datasets, comparisons with matched metadata budgets, and measured additional overhead relative to Sparse-Marlin. In particular, Table 16 shows that the gains are not merely attributable to an increased metadata budget.
Critical weaknesses

1.The main text’s claim of the “same storage budget” as dense 3-bit quantization remains insufficiently precise. The appendix itself reports that GapSQ requires 3.3125 bits per original weight with 4-bit quantization, 50% sparsity, and a group size of 128. Different formats also incur their own storage costs for scales, zero-points, masks, and actual packing overhead. Table 16 addresses metadata fairness among sparse methods, but does not fully substantiate the main text’s claim of a matched storage budget relative to the dense baselines. I would like to see actual model sizes reported in bytes, together with comparisons under matched actual storage budgets. 2.The connection between the key recoverability condition and the actual quantization pipeline is not sufficiently detailed. Section 4 checks branch separability using candidate quantization codes, but does not adequately explain whether this condition is still guaranteed after GPTQ error compensation and storage of the final parameters, or how groups with retained weights of only one sign or no retained weights are handled. Results under aggressive compression may be sensitive to the underlying configuration; for example, the 3-bit baseline PPL for Llama 3-3B is 359.47. Complete pseudocode, detailed quantization settings, and results from an appropriate number of repeated calibration runs would strengthen reproducibility.

Questions for authors
  1. Please report the actual model storage footprint, including masks, all metadata, and packing overhead, and revisit the claim of the “same storage budget” relative to the dense 3-bit baseline. Do the main conclusions regarding accuracy still hold when the actual storage sizes in bytes are strictly matched? The existing metadata-matched comparisons between sparse methods in Table 16 are helpful.
  2. At exactly which stage is the branch separability condition checked? Is this condition still guaranteed after GPTQ error propagation, FP16 storage of the shift magnitude, and final packing of the quantization codes? How are groups containing retained weights of only one sign, groups with no retained weights, and the zero-shift fallback handled? Please provide pseudocode corresponding to the main experiments.
  3. Please provide the damping, clipping, grouping, and calibration settings used for the severely degraded 3-bit baselines, and report results using a few differently sampled calibration sets. Do GapSQ’s substantial improvements primarily arise from robustly exploiting the gap, or do they depend on specific baseline configurations? These clarifications would affect my assessment of the strength of the reported gains.
6 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer FwBM (1/3)

We thank the reviewer for the time and effort spent evaluating our paper. Below, we address each concern in detail.

1. Storage-matched comparison (Question 1 and Weakness 1)

GapSQ remains effective under strictly aligned storage accounting, including weights, masks, and metadata. For the reviewer’s reference, we provide comparisons against dense and sparse baselines below, using group size 256 for GapSQ to align the total storage accounting.All results use Llama 3-8B. GPTQ baselines are from Tab. 2‾\underline{\text{Tab. 2}}; OmniQuant and AWQ baselines are from Appendix C, Tab. 11‾\underline{\text{Appendix C, Tab. 11}}. * denotes new experiments. Storage entries are in bits per original weight (bpw).

MethodGroup sizeWeight bpwMask bpwMetadata bpwTotal bpwWikiText-2 PPL ↓Avg. accuracy ↑
Dense GPTQ W3128300.163.1628.9765.4%
SparseGPT+GPTQ W4, 50% sparsity128210.163.1612.2466.0%
GapSQ W4, 50% sparsity*256210.163.1610.6267.0%
Dense OmniQuant W3128300.163.169.8066.7%
SparseGPT+OmniQuant W3, 33.3% sparsity128210.163.169.8967.7%
GapSQ–SparseGPT+OmniQuant W3, 33.3% sparsity*256210.163.169.6868.0%
Dense AWQ W3128300.163.169.5568.1%
SparseGPT+AWQ W3, 33.3% sparsity128210.163.169.5768.0%
GapSQ–SparseGPT+AWQ W3, 33.3% sparsity*256210.163.169.4768.3%

GapSQ still outperforms the corresponding dense and sparse baselines under the same storage accounting.

We have also corrected the imprecise “same storage budget” wording in Line 312‾\underline{\text{Line 312}}. Our intention was to match the weight-plus-mask bit-width while keeping the same quantization group size. Therefore, we also described shift-related metadata as additional group-level overhead in Sec. 4.4‾\underline{\text{Sec. 4.4}} for clarity, with detailed accounting in Appendix H‾\underline{\text{Appendix H}}.

AuthorsOpenReview ↗

Authors' Response to Reviewer FwBM (2/3)

2. Details of branch separability check (Question 2 and the first half of Weakness 2)

  • Integration with GPTQ

    For generality, separability is checked on the final group codes, after all within-group GPTQ error propagation. If these updates cause a weight to be quantized outside its original branch’s allowed code range, we discard that shift candidate, with the base quantizer with s=0s=0 retained as a fallback, as described in Line 269‾\underline{\text{Line 269}} and explained below. On Llama 3-8B at 50% sparsity with 4-bit GPTQ quantization, only 4.46% of groups finally use this fallback.

    After finalizing each group, its error is propagated only to later, unquantized groups. Thus, strict branch separability is preserved throughout compression for every group using a nonzero shift.

  • Special cases and zero-shift fallback

    Neither groups with no retained weights nor groups containing only one sign actually occurred in our main experiments. If encountered, both cases use the base quantizer with s=0s=0; empty groups reconstruct as zeros under the pruning mask.

    When s=0s=0, inverse shifting leaves the base quantizer’s output unchanged, enabling safe fallback. From Eq. (13)‾\underline{\text{Eq. (13)}},

    =D_{\mathrm{GPTQ}}(q_i).$$ The fallback therefore uses standard GPTQ codes and parameters with $s=0$. Here, any reference level is valid, and branch separation is unnecessary.
  • Pseudocode for details

    For clarity, we provide the following procedure. GPTQ_group includes within-group error propagation while keeping masked positions zero. Each candidate starts from the same saved group weights; later groups are updated only after the best candidate is selected. Here, * denotes elementwise multiplication.

    for each Linear:
        for each group with weights W_g and pruning mask M_g:
            W = M_g * copy(W_g)
            R_pos, R_neg = positive/negative retained indices of W
    
            best = GPTQ_group(copy(W), M_g)
            best.shift, best.reference = 0, 0
            best.error = output_MSE(W, M_g * decode(best), X_g)
    
            if the group has retained weights and both signs are present:
                for each nonzero shift candidate s:
                    U = M_g * (W - s * sign(W))
                    candidate = GPTQ_group(copy(U), M_g)
    
                    Q = candidate's final codes
                    r = min(Q[R_pos])
    
                    if max(Q[R_neg]) >= r:
                        continue
    
                    W_hat = M_g * inverse_shift(decode(candidate), s, r)
                    error = output_MSE(W, W_hat, X_g)
    
                    if error < best.error:
                        save candidate with shift s, reference r,
                            and error as best
    
            commit best
            propagate its group error to later, unquantized groups following the standard GPTQ procedure
    

    We will clarify these details in the revision.

AuthorsOpenReview ↗

Authors' Response to Reviewer FwBM (3/3)

3. Quantization settings and calibration robustness (Question 3 and the second half of Weakness 2)

  • Detailed GPTQ configuration

    For the Llama 3-3B baseline results highlighted by the reviewer (SparseGPT+GPTQ, 50% sparsity, 3-bit quantization, PPL 359.47), we used the damping, clipping-search, and block-size settings from the official GPTQ implementation for OPT. The configuration is listed below. Group size and calibration settings are already reported in Sec. 5‾\underline{\text{Sec. 5}}.

    SettingValue
    Damping0.01
    Clipping searchDisabled
    Quantization group size128
    GPTQ block size128
    Calibration datasetC4
    Calibration samples128
    Tokens per sample2048
  • Results with different calibration seeds and datasets

    We repeated the experiment with two additional C4 sampling seeds and also tested WikiText-2 calibration. Each row uses the same calibration data for both methods. All results below use Llama 3-3B at 50% sparsity with 3-bit quantization.

    Calibration settingSparseGPT+GPTQ PPL ↓GapSQ PPL ↓SparseGPT+GPTQ avg. accuracy ↑GapSQ avg. accuracy ↑
    C4, original run in Tab. 2359.4719.4847.6%57.8%
    C4, additional sampling seed A320.6019.2747.9%57.4%
    C4, additional sampling seed B336.7219.3347.4%57.9%
    WikiText-242.5616.8649.5%56.2%

    GapSQ improves both metrics in every tested setting. Thus, the gains are not limited to one particular sampling seed or to C4. For the reviewer’s reference, we also report GapSQ’s performance with C4, WikiText-2, and Pile calibration on Llama 3-8B in Appendix G, Tab. 15‾\underline{\text{Appendix G, Tab. 15}}.

  • Why are the 3-bit PPL gains particularly large?

    One favorable explanation is that low-bit quantization introduces larger errors, making gap handling more beneficial. With only eight levels at 3 bits, narrowing the pruning-induced gap reduces the represented range and allows finer quantization, which may help alleviate a quantization bottleneck under aggressive compression. This is consistent with Tab. 2‾\underline{\text{Tab. 2}}, where GapSQ achieves larger absolute PPL reductions at 3 bits than at 4 bits across all four models.


We hope this discussion clarifies the design and effectiveness of our method, resolves your main concerns, and encourages a positive reassessment of our work. We remain happy to answer any follow-up questions.

Reviewer_jj7GOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

After pruning, the remaining weights leave a gap around zero, so some quantization levels near zero are wasted. GapSQ shifts the positive and negative weights toward each other before quantization and shifts them back at dequantization. A constraint makes sure the two sides can still be told apart from their codes, so no extra sign bit per weight is needed, only some per-group metadata. The shift is chosen with a small calibration search. Experiments cover several Llama and Qwen models, different sparsity patterns and quantizers, plus kernel and serving measurements.

Critical strengths

The paper targets a specific, easy-to-understand inefficiency with a simple fix. The gap measurements make the motivation concrete. The branch-separation condition is the key trick: it lets the shift be undone without storing a sign bit for every kept weight. This is what makes the idea practical rather than just a better offline fit. The gains over the base compression pipeline are solid. They show up across model sizes, quantizers and sparsity patterns. On Llama 3-8B at 3 bits and 50% sparsity, average accuracy goes from 57.2% to 62.5%, and perplexity drops from 30.52 to 11.72 (Table 2). Appendix H compares against smaller group sizes with the same metadata budget, so the gains are not just from storing more metadata. Appendix E shows only a small serving overhead over Sparse-Marlin in the tested setup. The shift search seems easy to use without much tuning. In Table 6, four candidates already bring perplexity down from 12.239 to 10.596, almost the same as 10.578 with eight candidates. Trying 16 or 32 gives little extra benefit. This is useful because it suggests the method does not need a large search to work, and the extra fitting is done once during compression rather than during inference.

Critical weaknesses

1.The comparison with dense quantization is not clearly matched in actual storage. The main text says sparse 4-bit GapSQ and dense 3-bit GPTQ use the same storage. But Appendix H counts 3.3125 bits per original weight for GapSQ once the mask and metadata are included. Dense 3-bit also carries scale/zero metadata, and its total depends on group size, but the paper never shows the two totals side by side. The metadata-matched sparse baseline is a good control, but it answers a different question (sparse vs sparse). This matters most for Table 11, where the gains over dense AWQ and OmniQuant are small enough that a fraction of a bit could change the conclusion. 2.The setting with the best accuracy is not the setting that was deployed. The largest gains are for 3-bit unstructured sparsity, but the fast kernel is for 4-bit 2:4 sparsity. The end-to-end serving test is also small: 64 input tokens, 64 output tokens, QPS = 1. These numbers are useful, but they don't tell us the speed/quality trade-off for the configurations where GapSQ helps most. This doesn't take away from the gains over the matching baselines, but the practical claims should be scoped down. 3.The analysis does not fully show how much of the gain comes specifically from closing the pruning gap. Table 1 counts all unused quantization levels, including any outside the central gap, and the group visualizations show only a few examples. The search ablations change the number of shift candidates and the fitting loss, but do not compare against spending the same extra search budget on scale or clipping adjustments without shifting. That control would help show what the proposed shift adds beyond a better-tuned quantizer. It would also help to show whether groups with larger gaps actually benefit more across the model.

Questions for authors

Can you report actual packed model sizes (with all masks and metadata), and compare dense and sparse methods at the same total size?

Which accuracy configuration exactly matches the model used in the serving test? Is there an efficient kernel for the 3-bit configuration with the biggest gains, or a reason it is hard to build?

GPTQ updates weights sequentially to compensate errors. When is the branch-separation condition checked? Is it verified on the final stored codes, after every update that can touch the group?

4 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer jj7G (1/2)

We thank the reviewer for the time and effort spent evaluating our paper. Below, we address each concern in detail.

1. Storage-matched comparison (Question 1 and Weakness 1)

GapSQ remains effective under strictly aligned storage accounting, including weights, masks, and metadata. For the reviewer’s reference, we provide comparisons against dense and sparse baselines below, using group size 256 for GapSQ to align the total storage accounting.All results use Llama 3-8B. GPTQ baselines are from Tab. 2‾\underline{\text{Tab. 2}}; OmniQuant and AWQ baselines are from Appendix C, Tab. 11‾\underline{\text{Appendix C, Tab. 11}}. * denotes new experiments. Storage entries are in bits per original weight (bpw).

MethodGroup sizeWeight bpwMask bpwMetadata bpwTotal bpwWikiText-2 PPL ↓Avg. accuracy ↑
Dense GPTQ W3128300.163.1628.9765.4%
SparseGPT+GPTQ W4, 50% sparsity128210.163.1612.2466.0%
GapSQ W4, 50% sparsity*256210.163.1610.6267.0%
Dense OmniQuant W3128300.163.169.8066.7%
SparseGPT+OmniQuant W3, 33.3% sparsity128210.163.169.8967.7%
GapSQ–SparseGPT+OmniQuant W3, 33.3% sparsity*256210.163.169.6868.0%
Dense AWQ W3128300.163.169.5568.1%
SparseGPT+AWQ W3, 33.3% sparsity128210.163.169.5768.0%
GapSQ–SparseGPT+AWQ W3, 33.3% sparsity*256210.163.169.4768.3%

GapSQ still outperforms the corresponding dense and sparse baselines under the same storage accounting.

We have also corrected the imprecise “same storage budget” wording in Line 312‾\underline{\text{Line 312}}. Our intention was to match the weight-plus-mask bit-width while keeping the same quantization group size. Therefore, we also described shift-related metadata as additional group-level overhead in Sec. 4.4‾\underline{\text{Sec. 4.4}} for clarity, with detailed accounting in Appendix H‾\underline{\text{Appendix H}}.

2. Deployment and serving discussion (Question 2 and Weakness 2)

  • Matching accuracy configuration (Question 2)

    The serving test in Tab. 5, right‾\underline{\text{Tab. 5, right}} uses Llama 3-8B with 4-bit quantization and 2:4 sparsity, corresponding to the GapSQ row in Tab. 3‾\underline{\text{Tab. 3}}: 56.4% average accuracy and 18.66 WikiText-2 perplexity.

  • Why this deployment configuration? (Question 2 and Weakness 2)

    We chose this as a practical configuration for widely used NVIDIA GPUs, combining native 2:4 sparsity support, the optimized Sparse-Marlin W4 backend, and the mature vLLM serving framework. We also kindly note that using 2:4 sparsity for efficiency test is a common practice in pruning work including SparseGPT, Wanda or in joint compression work such as OBR.

  • 3-bit kernel support (Question 2)

    The Sparse-Marlin backend we used does not provide a kernel for 3-bit unstructured sparsity. Supporting this requires additional packing, decoding, and sparse-execution optimizations. Since our current work focuses on the gap-handling joint compression method, we leave broader kernel and platform support for future work.

  • Serving workload (Weakness 2)

    We kindly clarify that our original 64-input/64-output-token workload follows Sparse-Marlin’s evaluation setting. We additionally tested the default sequence lengths in vLLM’s random benchmark (1024 input tokens, 128 output tokens), keeping QPS=1. The results below use Llama 3-8B on NVIDIA A100 with W4 quantization and 2:4 sparsity.

    ModelTTFT (ms) ↓TPOT (ms) ↓E2EL (ms) ↓
    Dense FP16111.4412.131652.25
    GapSQ W4109.086.23900.72

    Under this longer workload, the TTFT benefit is smaller, while the decoding benefit is larger.

3. Branch-separation check with GPTQ (Question 3)

When combining GapSQ with GPTQ, we check branch separability on each candidate’s final group codes, after all within-group GPTQ error propagation. Candidates whose final codes violate separability are excluded from selection, while base GPTQ with s=0s=0 remains available as the fallback, as described in Line 269‾\underline{\text{Line 269}}. On Llama 3-8B with 50% sparsity and 4-bit quantization, only 4.46% of groups ultimately use this fallback.

After selecting and committing the best candidate, its group error is propagated only to later, unquantized groups, following the standard GPTQ procedure. The committed codes remain unchanged. Thus, the check uses the final stored codes after all updates that can affect the group, and subsequent error propagation cannot invalidate it.

AuthorsOpenReview ↗

Authors' Response to Reviewer jj7G (2/2)

4. Gap-specific analysis (Weakness 3)

  • Unused levels within the pruning-induced gap

    Here, we counted only the unused levels within the pruning-induced gap as suggested. The results below use Llama-3.1-8B at 50% sparsity with 4-bit quantization. The last three columns report the percentage of groups with at least 2, 3, or 4 unused levels within the gap.

    Pruning methodUnused levels in gap (%)Mean unused levels per groupGroups with ≥2 unused levelsGroups with ≥3 unused levelsGroups with ≥4 unused levels
    Magnitude18.63%2.98195.12%93.27%6.31%
    SparseGPT14.85%2.37780.76%54.32%3.31%
    Wanda12.85%2.05671.46%35.19%2.24%

    More than two of the 16 levels per group are unused within the gap on average. Groups with at least three unused levels are also common: they account for 54.32% of SparseGPT groups, with proportions ranging from 35.19% to 93.27% across the three pruning methods.

    We also kindly note that counting completely unused levels does not capture the full effect: levels near the branch boundaries may cover part of the empty gap without being completely unused. Shifting can therefore improve level assignment beyond making empty levels usable. It also reduces the represented range, which can allow finer quantization at the same bit-width, as discussed in Sec. 3‾\underline{\text{Sec. 3}}. We will revise the discussion to make these points clearer.

  • Comparison with matched-budget clipping search

    Thanks for the suggestion. We now provide a matched compression-time search budget: we added an 8-candidate clipping search to dense GPTQ W3 and SparseGPT+GPTQ W4, and compared them with GapSQ’s 8-candidate shift search (without additional clipping). All results use Llama 3-8B; sparse methods use 50% sparsity.

    MethodClipping searchWikiText-2 PPL ↓Avg. accuracy ↑
    Dense GPTQ W38 candidates13.2065.9%
    SparseGPT + GPTQ W48 candidates11.9766.3%
    GapSQ W4None10.6067.1%
    GapSQ W48 candidates10.3267.3%

    GapSQ still outperforms the baselines when they receive eight additional clipping candidates. Also, thanks to GapSQ's flexibility and generality, clipping search can also be added to GapSQ and further improves its results.

  • Do groups with larger gaps benefit more?

    Interesting question. To answer this, we analyzed all quantization groups across Llama 3-8B, using group size 128 at 50% sparsity with 4-bit quantization. We report the relative reduction in reconstruction MSE—the error discussed in Eq. (5)‾\underline{\text{Eq. (5)}} and the reduction estimated in Eq. (7)‾\underline{\text{Eq. (7)}}. We sorted groups by normalized gap size and divided them into ten groups of equal size (0–10% contains the smallest gaps, while 90–100% contains the largest gaps):

    Gap-size percentile0–10%10–20%20–30%30–40%40–50%50–60%60–70%70–80%80–90%90–100%
    Mean normalized gap0.12300.16780.18900.20500.21880.23180.24490.25920.27710.3115
    Mean relative MSE reduction10.05%18.19%21.84%24.10%25.93%27.92%30.43%33.72%37.87%42.70%

    Groups with larger gaps generally show larger improvements, with positive correlations of 0.5995 (Spearman) and 0.6121 (Pearson). This is consistent with the range-based estimate in Eq. (7)‾\underline{\text{Eq. (7)}}. Since GapSQ’s shift search also changes how weights are assigned to quantization levels and, in practice, minimizes activation-aware output MSE in Eq. (16)‾\underline{\text{Eq. (16)}}, a positive but not perfectly linear relationship is expected.

    The error reductions are noticeable: groups around the median gap size show average MSE reductions of approximately 26–28%. Gains of this magnitude are unlikely to arise simply from adding metadata or search candidates, as supported by our controls above. Moreover, GapSQ’s shift search is orthogonal to standard quantizer tuning strategies such as clipping search, allowing these optimizations to be applied alongside shifting. The further gains from combining them support GapSQ as an effective and flexible approach to handling pruning-induced gaps. We will add this useful analysis to the revised manuscript.


We hope this discussion clarifies the design and effectiveness of our method, resolves your main concerns, and encourages a positive reassessment of our work. We remain happy to answer any follow-up questions.

Thanks for your detailed rebuttal and experiments. However, I found it is very tricky. The authors listed the PPL and accuracy results on 3-bit, but deployment latency of 4-bit model. And the comparison of these models and precision are mismatch.

Also for 3.16 bpw setting, the gain is marginal, but the inference cost and real latency are not clear. So I will keep my initialized rating.

AuthorsOpenReview ↗

Follow-up to Reviewer jj7G: Clarification on Our Main Setting, Additional Experiments, and the Paper's Scope

We thank the reviewer for the feedback. However, we must clarify the following points to avoid potential misunderstandings.


1. Our deployment test uses a main 4-bit setting, not the 3-bit setting

4-bit quantization at 50% sparsity is a main setting in Tab. 2‾\underline{\text{Tab. 2}}, where GapSQ improves both PPL and average accuracy across all four models. This setting also (1) enables comparison with dense 3-bit GPTQ at a matched weight-plus-mask bit-width, which the sparse 3-bit setting cannot offer, and (2) provides better absolute quality than the corresponding 3-bit setting under the same sparsity pattern.

There is also a practical implementation reason for this choice. As explained in our previous response, Sparse-Marlin is designed for 4-bit quantized weights with 2:4 sparsity, providing an optimized deployment backend in vLLM. Using hardware-supported 2:4 sparsity for GPU evaluation is also established practice in SparseGPT, Wanda, and OBR. These quality and implementation considerations make W4 a well-supported deployment choice.

Reporting larger relative gains against SparseSPT+GPTQ at 3 bits in Tab. 2‾\underline{\text{Tab. 2}} does not necessarily make 3-bit a more suitable deployment target.

2. Purpose of the storage-matched 3.16 bpw comparison

We added these experiments to address the reviewer’s original storage concern: whether shifting still improves quality when the total storage for weights, masks, and all metadata is matched. They are additional storage controls and do not replace our main experiments. The comparisons in Tab. 2‾\underline{\text{Tab. 2}} keep the base quantization settings fixed, including group size. The extra shift metadata is described in Sec. 4.4‾\underline{\text{Sec. 4.4}} and counted in Appendix H‾\underline{\text{Appendix H}}. The additional group-size-256 results show that GapSQ still improves on the corresponding joint-compression baselines under matched storage accounting.

The AWQ and OmniQuant comparisons extend the results in Appendix C‾\underline{\text{Appendix C}}, which originally intended to show that GapSQ can benefit other modern quantizers. Testing latency for these additional formats (as now pointed out by the reviewer) would address a separate deployment question and require substantial implementation work. Such work falls outside the scope of the originally requested storage controls and the main focus of our paper, and would require more time than is available during this author–reviewer discussion period. We would also like to note that this does not change the answer to the original storage question: the quality gains remain after all storage overhead is counted.


We also emphasize that neither point contradicts our main claim: addressing the pruning-induced gap improves quality in prune-then-quantize joint compression pipelines. As our study is method-centric rather than focused on system-level deployment, extending the implementation to additional formats and inference backends lies outside our current scope and is left for future work. We respectfully ask that our core contribution be fairly assessed against the reported evidence, with the purpose and scope of each experiment taken into account.

We remain open to follow-up discussion.

Reviewer_PYBoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce the challenges associated with a joint sparse and quantized compression. They clearly explain why a simply application of a mature pruning solver followed by a quantization solver introduces large errors and propose a method to overcome these challenges that is agnostic to the used pruning/quantization solver. The proposed method yields large gains compared to a naive sequential composition of the above.

Critical strengths

1/ The paper is easy to understand and well-written

2/ The proposed method seems to be quite easy to implement and is pruning/quantization solver agnostic

3/ The gains compared to a simple composition of methods are large.

Critical weaknesses

1/ The method is built around a prune-then-quantize setting. There is limited evaluation compared to a quantize-then-prune approaches.

2/ I wonder if non-uniform quantizers and vector-quantization schemes like Quip# and QTIP can be incorporated here. It is less intuitive that shifting helps in that setting.

3/ There seems to be another method for joint quantization and pruning not discussed in the paper JSQ [1]. Can the authors highlight the differences of this approach. Whether it can be incorporated within the proposed framework or if it can be included in some comparisons?

[1] Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer PYBo

We thank the reviewer for the time and effort spent evaluating our paper. Below, we address each concern in detail.

1. Additional comparison with quantize-then-prune

For the reviewer’s reference, we expanded the existing discussion of compression order in Appendix F‾\underline{\text{Appendix F}} with additional comparisons on Llama 3-8B and Qwen2.5-7B at 50% sparsity with 3- and 4-bit quantization, using SparseGPT+GPTQ. Each cell reports WikiText-2 PPL (↓) / six-task average accuracy (↑).

Order/methodLlama 3-8B, 4-bitLlama 3-8B, 3-bitQwen2.5-7B, 4-bitQwen2.5-7B, 3-bit
Quantize→Prune30.16 / 64.3%36.43 / 56.1%11.18 / 68.4%13.26 / 63.9%
Prune→Quantize12.24 / 66.0%30.52 / 57.2%10.85 / 68.8%12.69 / 65.8%
GapSQ10.58 / 67.1%11.72 / 62.5%10.34 / 70.3%11.81 / 67.0%

Prune→Quantize and GapSQ results are from Tab. 2‾\underline{\text{Tab. 2}}. The Llama 3-8B 4-bit Quantize→Prune result is from Appendix F, Tab. 14‾\underline{\text{Appendix F, Tab. 14}}; the remaining Quantize→Prune results are new.

Prune-then-quantize performs better than quantize-then-prune in both metrics across all four settings. GapSQ builds on this pipeline and further improves both PPL and accuracy by addressing the pruning-induced gap. We will include these expanded comparisons in Appendix F‾\underline{\text{Appendix F}}.

2. Compatibility with non-uniform quantization

We expect GapSQ to also help non-uniform quantizers when a pruning-induced gap remains in the representation being quantized and is not explicitly accounted for. GapSQ's shifting would still adapt the post-pruning weight distribution and thus can improve its match to the quantizer’s codebook.

As shown below, we conducted additional experiments with SparseGPT+QuIP# at 50% sparsity with 4-/3-bit quantization. Adding GapSQ yields clear improvements in both WikiText-2 perplexity and average accuracy:

Weight bitsMethodWikiText-2 PPL ↓Avg. accuracy ↑
4SparseGPT+QuIP#11.617365.45%
4GapSQ-SparseGPT+QuIP#11.280365.90%
3SparseGPT+QuIP#12.616963.67%
3GapSQ-SparseGPT+QuIP#12.374564.20%

3. Other joint compression work

Thank you for suggesting this work. We kindly clarify the main differences here and will add it to the revision.

The cited work [1] requires hypernetwork training (10,000 steps) and jointly learns structured channel pruning and mixed-precision policies. GapSQ uses lightweight calibration without training and focuses on unstructured and semi-structured weight pruning with a uniform weight bit-width.

These differences in optimization cost and pruning structure make direct comparison and integration less straightforward under our current setting. We leave further investigation of [1]-style settings to future work.

[1] La, Hoang-Loc, et al. "Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression." arXiv preprint arXiv:2606.07819 (2026).


We hope our response and discussion fully address your comments, and we remain open to any follow-up questions.

To Reviewer PYBo,

Thank you again for your positive feedback. As the discussion period is ending soon, could you please let us know if our response has addressed your remaining questions?

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.