acceptodds
Under review as a conference paper at ICLR 2027

QUENCH: Prune Iteratively and Quantise Once for Training-Free LLM Compression

Abstract

Large language models require substantial memory to store their weights. Pruning removes selected weights and quantisation represents them with fewer bits, but both can reduce accuracy. A small low-rank correction can recover some of the lost accuracy without fine-tuning. We introduce QUENCH, a training-free pipeline that designs pruning and quantisation around such a correction. We target the linear layers of the transformer architecture. QUENCH alternates pruning and activation-aware correction, enabling the sparse weights to respond to the previous correction. It then quantises the surviving weights once with elastic-bucket quantisation (EBQ), which learns representative weight values (codebooks) for each output channel, and refits the correction. On LLaMA-3-8B with 2:4 sparsity and 3-bit values, QUENCH reaches WikiText2 perplexity 12.01 and 34.22% ARC-Challenge accuracy, against 12.60 and 31.22% reported for EoRA with 2:4 sparsity and 4-bit weights. We also weigh the improvement from activation weighting against low-rank correction. It improves standalone codebooks by 12.29 points, but provides no consistent additional benefit after a rank-128 EoRA correction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model compression, post-training quantization, pruning, low-rank approximation, quantization error compensation, semi-structured sparsity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4jCkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a joint low-rank + sparsity + quantization weight compression strategy, where low-rank and sparsity factors are optimized via alternating optimization (Joint-SLR). Then one quantizes the sparse factor via EBQ (non-uniform quantization) and refits the final low-rank correction residual. The approach is validated on the Llama-2/Llama-3 model family on standard LM evals.

Critical strengths
  • The proposed method achieves a quite strong quality/compression trade-off, consistently outperforming baselines from the literature. The closest competitor method, ApiQ [12], requires training, whereas Quench is training-free.
  • The design choices of Quench—such as the choice of quantizer, order of the quantization operation, and number of iterations—are carefully ablated.
Critical weaknesses
  • The idea lacks conceptual novelty. The ideas of alternating sparse + low-rank [1, 2, 3, 4], quantized + low-rank [5, 6, 7], and sparse + quantized [8, 9] appeared in a line of work. The main contribution appears to be the combination of existing techniques and the right choice of order.
  • Evaluation involves only outdated models; Llama-2 and Llama-3 were released 3 and 2 years ago, respectively. Evaluation on more recent models (Qwen-3.x, Gemma-4) and reasoning tasks (GPQA-Diamond, AIME, MATH500) or agentic tasks would significantly strengthen the validation.
  • There is a related work, ProjQ [10], claiming to achieve strong compression results (low-bit quantization + low-rank) with a moderate accuracy drop. I think it is worth being added to the comparison as a baseline. OBR [11] is also a natural baseline for the sparse + quantized setup.

[1] Stephen Zhang and Vardan Papyan. 2024. OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition. arXiv:2409.13652. Google Scholar.

[2] Mehdi Makni, Kayhan Behdin, Zheng Xu, Natalia Ponomareva, and Rahul Mazumder. 2025. HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs. arXiv:2502.00899. Google Scholar.

[3] Mehdi Makni, Xiang Meng, and Rahul Mazumder. 2025. 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs. Advances in Neural Information Processing Systems, 38. Google Scholar.

[4] Zeliang Zong, Kai Zhang, Zheyang Li, Wenming Tan, Ye Ren, Yiyan Zhai, and Jilin Hu. 2025. 1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models. Findings of EMNLP 2025, pp. 14206–14220. Google Scholar.

[5] Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, and Mert Pilanci. 2024. Compressing Large Language Models using Low Rank and Low Precision Decomposition. NeurIPS 2024. Google Scholar.

[6] Yoonjun Cho, Soeun Kim, Dongjae Jeon, Kyelim Lee, Beomsoo Lee, and Albert No. 2025. Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition. Findings of ACL 2025, pp. 14453–14470. Google Scholar.

[7] Hyochan Chong, Dongkyu Kim, Changdong Kim, and Minseop Choi. 2026. NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models. arXiv:2602.06694. Google Scholar.

[8] Jing Liu, Toshiaki Koike-Akino, Ye Wang, Hassan Mansour, and Matthew Brand. 2025. AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descent. arXiv:2506.10205. Google Scholar.

[9] Xianglong Yan, Tianao Zhang, Zhiteng Li, Haotong Qin, and Yulun Zhang. 2025. Progressive Binarization with Semi-Structured Pruning for LLMs. arXiv:2502.01705. Google Scholar.

[10] Wenya Yu, Chao Zhang, Li Wang, Samson Lasaulce, and Merouane Debbah. 2026. ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression. arXiv:2606.00494. Google Scholar.

[11] Hang Guo, Yawei Li, and Luca Benini. 2025. Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs. arXiv:2509.11177. Google Scholar.

Questions for authors
  • How well does the method perform for lower bit widths (1–2-bit quantization)?
  • How significant is the decoding cost for the sparse-quantized + low-rank representation? Could it significantly slow down the inference speed?
  • Does the choice of SparseGPT [13] as a pruning algorithm significantly affect the result? What if one uses magnitude pruning / Wanda [14] instead?
  • How long does it take to compress the Llama-3-8B model via QUENCH, for instance?

[12] Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. 2024. ApiQ: Finetuning of 2-Bit Quantized Large Language Model. EMNLP 2024, pp. 20996–21020. Google Scholar.

[13] Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. arXiv:2301.00774. Google Scholar.

[14] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A Simple and Effective Pruning Approach for Large Language Models. arXiv:2306.11695. Google Scholar.

3 comments

Thank you for the review. Tables G1 and G2 are in the general response.

Novelty: combination and order. We agree that the components exist. The revised paper states the contribution as a rule for combining them, which prior work does not establish: quantise once after an iterative sparse + low-rank stage, then refit the correction.

Existing methods handle quantisation differently:

  • HASSLE-free, 3BASiL and OATS do not quantise.
  • CALDERA quantises inside its alternating loop.
  • SLiM quantises first.

Our results show that the order matters:

  • Table G1: quantising last gives 12.09 vs 16.74 PPL over three seeds.
  • Table G2: the same rule takes existing sparse + low-rank methods to 3 bits and beats EoRA 4-bit with 0.49 fewer bits per weight.

Recent models and reasoning tasks. We added Qwen3-8B-Base; the full table is in our reply to ivHt. With 3BASiL as stage 1, QUENCH beats EoRA 2:4 + 4-bit on all four benchmarks at lower storage. This includes the reasoning benchmarks GSM8K (25.02 vs 20.92) and MathQA (36.95 vs 35.04). Long-generation benchmarks (GPQA-Diamond, AIME, MATH500) did not fit in the compute available during the discussion period.

ProjQ and OBR.

  • OBR jointly quantises and sparsifies, without a low-rank branch.
  • ProjQ combines quantisation with a low-rank adapter, without sparsity.

Each covers two of the three components. The question we study, when to quantise relative to an iterative sparse + low-rank stage, arises only when all three are present. We have added both to the related work.

Q1. Lower bit widths. QUENCH with 3BASiL as stage 1, PPL / ARC-C:

  • 4-bit: 8.73 / 37.80
  • 3-bit: 10.06 / 33.19
  • 2-bit: 29.44 / 19.37

3-bit is the lowest width at which the training-free recipe preserves accuracy. At 2 bits, methods with training, such as ApiQ, remain necessary.

Q2. Decoding cost. Inference weight memory falls from 15.1 to 10.0 GiB. With public kernels, prefill throughput is 95% of dense, and batch-1 decoding takes 36.8 ms per token against 24.7 ms. The overhead comes from the 2:4 kernel at small batch sizes and from the separate low-rank product, and a fused kernel would remove it. Full numbers are in our reply to ivHt.

Q3. Choice of pruner. QUENCH perplexity with each pruner inside the loop:

Pruner1 round5 rounds
SparseGPT13.3212.24
Wanda17.4416.15
Magnitude18.9115.96

Iterating helps every pruner. SparseGPT stays about 4 points ahead because it also updates the kept weights, while Wanda and magnitude only select the mask.

Q4. Compression time for LLaMA-3-8B. 3.9 h for five rounds on one GPU (1.2 h for one round), with peak GPU memory of 16.3 GiB.

Thank you for your response. The revised framing better reflects the scope of the paper, and the additional experiments with Qwen3 models and low-bit compression provide further evidence of the method’s effectiveness.

After considering the rebuttal, including your responses to my comments and those of the other reviewers, I have decided to raise my score.

For the final revision, I suggest stating the paper’s contributions more explicitly in the main text and giving greater prominence to its key findings.

Thank you for raising your score and for the suggestion. We have updated the revision accordingly. Each results subsection of Section 5 now opens with the finding it supports, and the contributions in the introduction each point to their evidence.

Reviewer_ivHtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QUENCH, a training-free LLM compression framework that combines structured pruning, non-uniform quantization, and activation-aware low-rank correction. Its Joint-SLR component iteratively alternates SparseGPT-based pruning and EoRA-based correction, allowing the sparse weights to adapt to the preceding correction. The framework then applies Elastic-Bucket Quantization (EBQ) once to the remaining weights, followed by a final low-rank correction. Experiments on LLaMA-2-7B and LLaMA-3-8B demonstrate improvements in model accuracy and perplexity, showing that quantizing after iterative pruning outperforms quantizing within each iteration. The authors also find that activation-weighted quantization provides limited additional benefits when combined with EoRA correction.

Critical strengths
  1. A practically relevant problem and a coherent compression pipeline.

The paper addresses an important problem in post-training LLM compression: how pruning, quantization, and low-rank error correction should interact to preserve model quality under aggressive compression. Rather than independently improving individual compression components, QUENCH considers their ordering and mutual dependencies. The resulting pipeline is conceptually straightforward and does not require gradient-based fine-tuning.

  1. Meaningful empirical improvements from iterative pruning and correction.

The Joint-SLR experiments provide evidence that revisiting sparse weights after low-rank correction can improve model quality without increasing the final correction rank. On LLaMA-3-8B, increasing the number of pruning iterations from one to five improves ARC-Challenge accuracy from 35.15% to 40.02% and reduces WikiText2 perplexity from 10.91 to 9.92. The corresponding improvements on LLaMA-2-7B further support the usefulness of iterative reconstruction in the evaluated settings.

  1. An informative analysis of the interaction between quantization and low-rank correction.

One particularly interesting observation is that activation-aware weighting and low-rank correction do not necessarily provide additive benefits. In the 3-bit quantization experiments, activation weighting improves standalone EBQ by 12.29 percentage points on ARC-Challenge, whereas unweighted EBQ performs comparably after rank-128 EoRA correction. This finding suggests that different compression components may compensate for overlapping sources of reconstruction error, providing useful insight into the design of compression pipelines.

Critical weaknesses
  1. Limited methodological novelty relative to existing sparse-low-rank optimization and learned-codebook quantization methods.

The main components of QUENCH appear to have substantial overlap with existing literature. In particular, the proposed Joint-SLR procedure closely resembles prior alternating sparse-plus-low-rank optimization frameworks, such as OATS, HASSLE-free, and 3BASiL. HASSLE-free already alternates between sparse reconstruction and low-rank optimization while allowing the sparse support to change across iterations. Moreover, 3BASiL investigates a closed-form low-rank update for its improved HASSLE-free baseline, which is closely related to the activation-aware reconstruction used in EoRA.

Similarly, the proposed Elastic-Bucket Quantization (EBQ) is based on per-channel Lloyd quantization with min-max initialization and optional importance weighting, which largely overlaps with existing learned-codebook quantization approaches, particularly LeanQuant. Therefore, the methodological contribution appears to lie primarily in combining existing techniques and designing their execution schedule, rather than introducing fundamentally new pruning or quantization algorithms.

Although the overall pipeline produces encouraging results, the paper does not sufficiently distinguish its technical contributions from these closely related methods.

  1. Insufficient comparisons with the strongest related compression methods under consistent experimental settings.

A substantial portion of the experimental comparisons relies on previously published results for fixed SparseGPT- or GPTQ-compressed models followed by different low-rank correction methods. However, QUENCH modifies the compressed weight representation itself through iterative pruning and learned-codebook quantization. Therefore, comparisons against methods that only reconstruct errors from fixed compressed weights do not fully establish the advantages of the proposed framework.

In particular, direct end-to-end comparisons with methods that jointly optimize sparse and low-rank representations, such as OATS, HASSLE-free, and 3BASiL, are missing. Furthermore, some reported comparisons involve different quantization granularities, correction ranks, or effective storage budgets, making it difficult to attribute the observed improvements exclusively to QUENCH's design.

  1. Limited experimental scope and task-specific calibration weaken the generality of the conclusions.

The experiments are limited to LLaMA-2-7B and LLaMA-3-8B, which are relatively small models from closely related families. It remains unclear whether the proposed iterative compression strategy generalizes to larger models or different architectures.

  1. Practical compression efficiency and deployment benefits are not sufficiently demonstrated.

Although QUENCH is motivated by memory-efficient LLM deployment, its efficiency analysis primarily relies on theoretical storage estimates based on ideal packed representations. The paper does not report actual inference-time memory consumption, throughput, or latency.

This is particularly important because QUENCH combines structured sparsity, non-uniform per-channel codebooks, and an additional low-rank correction branch. These components may introduce nontrivial runtime overhead and require specialized implementations to achieve their theoretical efficiency benefits.

Furthermore, iterative pruning and correction require multiple calibration and reconstruction passes, potentially increasing the computational cost of compression compared with one-shot alternatives. However, the paper does not provide a systematic analysis of preprocessing time or the trade-off between additional iterations and their resulting accuracy improvements.

Questions for authors
  1. Could the authors clarify the specific methodological differences between Joint-SLR and existing alternating sparse-low-rank optimization methods, particularly HASSLE-free and 3BASiL? Direct comparisons under matched sparsity, correction rank, and calibration settings would help establish the novelty of Joint-SLR. Additionally, comparing EBQ with LeanQuant-style quantization under the same EoRA correction would help isolate the contribution of the proposed quantization component.

  2. Could the authors provide direct end-to-end comparisons with stronger joint compression baselines, particularly HASSLE-free, 3BASiL, and SLiM, under consistent evaluation protocols? Comparisons with matched effective bits per weight, correction rank, calibration data, and compression-time computational budgets would help determine whether QUENCH provides a meaningful advantage over existing approaches.

  3. Could the authors extend the evaluation to larger models and additional model families to demonstrate the generalizability of QUENCH?

  4. Could the authors provide a more comprehensive analysis of the practical efficiency of QUENCH, including actual GPU memory consumption, inference latency, throughput, and total compression time? In particular, evaluating how compression costs scale with the number of pruning iterations and reporting accuracy–memory–latency trade-offs against representative baselines would help determine whether the theoretical compression benefits translate into practical deployment advantages.

3 comments

Thank you for the review. Tables G1 and G2 are in the general response.

W1 / Q1. Differences from HASSLE-free and 3BASiL; EBQ vs LeanQuant under the same correction. The differences between Joint-SLR and HASSLE-free are small. Each Joint-SLR round is a full pass over the model, with activation statistics recomputed from the updated model. HASSLE-free iterates within a layer with fixed statistics. Under matched sparsity, rank and calibration, Joint-SLR is close to HASSLE-free at a similar budget and below 3BASiL (table in our reply to F2vq). We therefore do not claim Joint-SLR as the contribution. The revised paper presents QUENCH as a quantisation schedule that applies to any of these methods (Tables G1–G2).

EBQ vs LeanQuant under the same EoRA correction (dense 3-bit, rank 64, same calibration; WikiText2 PPL):

QuantiserNo correctionWith EoRA
LeanQuant (weighted, error feedback)7.707.31
EBQ, unweighted22.807.42
EBQ, activation-weighted–7.41

The correction reduces a 15.1-point gap to 0.1. ARC-C and MathQA differ by less than one point (LeanQuant + EoRA 47.44 / 35.58; EBQ + EoRA 46.67 / 34.61). Once the correction is in place, the choice of quantiser has little effect, so the simplest quantiser is sufficient.

W2 / Q2. End-to-end baselines under one protocol. All baselines now run end to end with our evaluation code, at 2:4 sparsity, rank 128 and the same calibration data: EoRA 2:4 + 4-bit, HASSLE-free and 3BASiL (Table G2), and SLiM. SLiM matched to our bit width and rank:

Methodb/wPPLARC-C
SLiM, 3-bit, rank 1283.2744.8322.61
SLiM, 4-bit, rank 1283.7723.4329.44
QUENCH3.2912.0130.29
  • Check on our SLiM run: it reproduces the published LLaMA-2-7B result (7.73 vs 7.77).
  • Why SLiM falls behind: it quantises before pruning, and the gap is consistent with our first finding.
  • Compression time on one GPU: SLiM 0.3 h, QUENCH 3.9 h, QUENCH with 3BASiL 5.7 h, and HASSLE-free at 80 iterations 7.8 h.

W3 / Q3. Other model families and larger models. We added Qwen3-8B-Base, under the same protocol:

Methodb/wPPLARC-CMathQAGSM8K
Dense167.0052.5655.5472.18
EoRA, 2:4 + 4-bit3.819.6538.2335.0420.92
QUENCH (stage 1: Joint-SLR)3.339.7937.3732.5318.73
QUENCH (stage 1: 3BASiL)3.339.2244.3736.9525.02

With 3BASiL as stage 1, QUENCH beats EoRA 4-bit on all four benchmarks with 0.48 fewer bits per weight.

On scale: QUENCH processes one transformer block at a time, so its peak GPU memory depends on block size, not model size (16.3 GiB for LLaMA-3-8B). Within the discussion period we prioritised a newer model family over larger models.

W4 / Q4. Efficiency. Measured on one L40S GPU for LLaMA-3-8B.

  • Compression:
    • 1.2 h for one round and 3.9 h for five, so each extra round adds about 40 minutes.
    • Perplexity improves from 13.14 to 12.01 over the five rounds (same GPU).
    • Peak GPU memory is 16.3 GiB, so a 24 GB card is sufficient.
  • Memory at inference:
    • Weights take 10.0 GiB, against 15.1 GiB dense, and peak memory is 13.8 GiB against 18.9 GiB. Both are measured with 2:4 weights held in 16-bit.
    • Packed 3-bit storage is 4.98 GB, against 16.06 GB.
  • Speed, with public kernels (CUTLASS 2:4; no fused kernel for the codebooks or the low-rank branch):
    • Prefill throughput is 95% of dense (11.2k vs 11.8k tokens/s at 2048 tokens).
    • Decoding at batch size 1 takes 36.8 ms per token against 24.7 ms. At this size the 2:4 kernel is inefficient, and the low-rank branch runs as a separate matrix product.

A fused kernel would remove this overhead. With current kernels, the practical benefit of QUENCH is memory.

Reviewer_ivHtOpenReview ↗

Reply to rebuttal

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns, particularly through the improved baseline comparisons and additional analyses. Most of my original questions have been satisfactorily addressed. However, two key concerns remain:

1. Practical efficiency. Although the additional measurements demonstrate memory savings, QUENCH incurs substantially higher decoding latency than the dense model (36.8 ms vs. 24.7 ms per token), alongside increased compression costs. Moreover, the reported 4.98 GB packed 3-bit storage has not been validated as an actual end-to-end inference memory footprint, as the inference measurements use 2:4 sparse weights stored in 16-bit precision. While the authors suggest that fused kernels could eliminate the runtime overhead, this remains unverified without an actual implementation and empirical evaluation. Therefore, the practical deployment benefits of QUENCH remain insufficiently demonstrated.

2. Scalability and generalizability. While I appreciate the additional experiments on Qwen3-8B, the evaluation remains limited to 7B–8B models. I believe that experiments on at least one substantially larger model, such as a 13B- or 30B-scale model, are necessary to provide more convincing evidence of the framework's scalability and generalizability.

Overall, while the rebuttal has addressed most of my concerns, these remaining issues regarding practical deployment efficiency and large-scale validation prevent me from increasing my assessment. I therefore maintain my initial score of 2.

Thank you for the follow-up. We answer the two remaining points with our own measurements.

1. Practical efficiency

Measured versus computed. Measured on one L40S at batch size 1, with the 2:4 weights held in 16 bits: weight memory falls from 15.1 to 10.0 GiB, and peak memory from 18.9 to 13.8 GiB. The 4.98 GB packed size is computed from the storage formula. The revision now labels it as computed.

The measured decode time reflects the kernels, not the format. Batch-1 decoding is limited by memory traffic. Our dense model decodes at 24.7 ms per token while reading 15.1 GiB of weights, an effective 655 GB/s.

  • At that rate, reading the measured 2:4 format (10.0 GiB) would take 16.4 ms. The measured 36.8 ms is 2.2× slower than its own data volume allows. The excess comes from the kernels we used: the CUTLASS 2:4 kernel is designed for large matrix products, and the low-rank branch runs as a separate product.
  • At the same rate, the packed QUENCH format would take about 7.6 ms per token. This is a bandwidth estimate.

Each component already has a public kernel.

  • SqueezeLLM serves per-channel non-uniform 3-bit codebooks, the format EBQ produces, with lookup-table kernels, and reports up to 2.3× faster decoding than FP16.
  • Sparse-MARLIN serves 2:4-sparse 4-bit weights and reports up to 3.2× end-to-end speedup over 16-bit serving in vLLM.
  • The rank-128 branch adds 0.77 bits per weight, about 5% of the dense weight traffic.

A single kernel combining the three is engineering work we have not done, and the revision states this.

The same applies to the baselines. EoRA with 2:4 sparsity and 4-bit weights, and SLiM, have the same runtime structure as QUENCH: a 2:4 base with quantised values plus a low-rank branch. Any kernel that serves them also serves QUENCH. QUENCH reads 13% fewer bytes than EoRA (3.29 against 3.78 bits per weight), with lower perplexity and higher ARC-C and MathQA.

Compression cost. Compression is a one-time cost: 1.2 h for one round and 3.9 h for five, on one GPU with under 24 GB of memory. This is less than the stronger first stages we compare with, 3BASiL (5.2 h) and HASSLE-free at 80 iterations (7.8 h).

2. Scale

  • Three architectures, the same behaviour. We use LLaMA-2-7B (multi-head attention), LLaMA-3-8B (grouped-query attention, 128k vocabulary) and Qwen3-8B (grouped-query attention with query-key normalisation). On all three, perplexity improves from one to five rounds: 9.29 → 8.26, 13.14 → 12.01 and 10.29 → 9.79.
  • Nothing in the procedure depends on model size. Each step (SparseGPT, the per-row codebooks, the EoRA refit) acts on one linear layer with its own calibration statistics. Blocks are processed in sequence, so peak memory is set by one block and time grows linearly with the number of blocks.
  • Larger models are the easier case. SparseGPT reports that larger models lose less accuracy at the same sparsity, which suggests that 7–8B models are the harder setting for 2:4 sparsity with 3-bit values.

A 13B run did not fit in the compute available to us during the discussion period: we work on a shared academic cluster, where GPU jobs often waited many hours to start. We will add a 13B model under the same protocol to the final version.

In short, the open question on speed is a kernel question shared by every method that combines 2:4 sparsity, quantisation and a low-rank branch, including EoRA and SLiM. The paper's claims concern accuracy at a given storage, which we measure under one protocol.

Reviewer_F2vqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QUENCH, a training-free LLM compression pipeline that combines iterative semi-structured (2:4) pruning, low-rank error correction, and quantization. The method alternates between SparseGPT pruning and EoRA correction, then it quantizes the weights using learned per-row codebooks and refits the correction. Experiments on LLaMA-2-7B and LLaMA-3-8B show improvements over single-pass pruning and demonstrate that quantizing after the iterative procedure performs substantially better than quantizing within each iteration.

Critical strengths

The paper addresses a relevant question about how pruning, quantization, and low-rank correction should interact when a corrector follows compression. The comparison of quantization schedules is informative, since the two schedules coincide at a single round, and the activation-weighting ablation provides a useful observation about how the codebook and the corrector split the work. The paper is generally clear, with well-organized experiments and reasonable disclosure of evaluation settings.

Critical weaknesses
  1. Limited novelty over existing sparse-plus-low-rank methods. Joint-SLR (Eq. 4) closely follows the alternating reconstruction framework of HASSLE-free. Besides, 3BASiL already evaluates alternating SparseGPT with closed-form weighted-SVD updates as a baseline. The paper does not compare against either approach, making it difficult to establish the contribution of Joint-SLR beyond existing methods. The effects of iterative recalibration and sparse support updates are also not isolated.

  2. Insufficient evidence for the advantage of the full pipeline. The strongest results come from dense EBQ with EoRA rather than QUENCH. Dense EBQ achieves 45.73 ARC-C and 7.42 PPL at 3.41 bits/weight, compared with QUENCH's 34.22 and 12.01 at an estimated 3.0–3.3 bits/weight. Without storage-matched comparisons or inference measurements, the benefit of structured pruning remains unclear. Furthermore, several baselines are evaluated only as corrections to fixed GPTQ weights rather than as complete compression methods. The use of task-specific calibration, which may also affect the sparse support, further limits the generality of the reported results.

Overall, I lean toward rejection. The paper offers a useful observation about when to quantize relative to low-rank correction, but its central iterative procedure closely follows an existing alternating scheme and is never compared against it directly. The experiments also do not show that the full pipeline improves on dense quantization with correction at a comparable storage budget.

Questions for authors
  1. How does Joint-SLR compare with HASSLE-free/3BASiL under matched settings? How much of the improvement comes from support updates versus recalibration?
  2. What is QUENCH's total storage cost including sparse metadata, and how does it compare with dense EBQ + EoRA at matched budgets?
  3. Are the sparse weights shared across tasks, and how does a single task-independent checkpoint perform?
2 comments

Thank you for the review. Tables G1 and G2 are in the general response.

W1 / Q1. HASSLE-free and 3BASiL under matched settings; support updates vs recalibration. We ran both methods from the authors' code, at 2:4 sparsity, rank 128 and the same WikiText2 calibration as Joint-SLR. Before quantisation:

Stage 1PPLARC-CCompression time
Joint-SLR, 5 rounds9.9236.353.3 h
HASSLE-free, 5 it.9.7937.032.5 h
HASSLE-free, 80 it.8.8840.707.8 h
3BASiL8.1439.515.2 h

Joint-SLR is close to HASSLE-free at a similar budget, and 3BASiL is stronger. We therefore treat Joint-SLR as one valid stage 1 and place the contribution in the quantisation schedule. Table G2 applies the schedule to each method: a stronger stage 1 gives a better compressed model, and every variant beats EoRA 4-bit at lower storage.

To separate support updates from recalibration, we fix the mask after round 1. The remaining rounds then recalibrate only the kept weights and the correction.

Joint-SLRPPLARC-C
1 round10.9132.42
5 rounds, mask fixed after round 110.4734.04
5 rounds, mask updated9.9236.35
  • Size of the effect: mask updates account for 56–64% of the improvement in perplexity, ARC-C and layer reconstruction error.
  • How much the mask moves: 7.2% of mask positions change in round 2, and 4.4% in round 5.
  • After quantisation the effect is larger: with a fixed mask, QUENCH keeps none of the ARC-C gain (27.73, against 27.82 after one round and 30.29 with mask updates).

The correction helps mainly by moving the support. This is why the support should settle before the values are quantised.

W2 / Q2. Storage and dense quantisation. QUENCH stores 3.29 bits per weight. This includes 1 bit per weight of 2:4 metadata, the per-row codebooks and the 16-bit low-rank factors.

Dense EBQ + EoRA (rank 64, 3.41 b/w) reaches lower perplexity at similar storage. The two methods serve different settings. QUENCH targets deployments that use 2:4 structure, where it reduces measured weight memory at inference from 15.1 to 10.0 GiB. Within that setting, it outperforms EoRA and SLiM at comparable or lower storage (Table G2, and SLiM in our reply to ivHt). The revised discussion (Section 6) states this scope.

On baselines that only correct fixed GPTQ weights: every comparison in the revision is now an end-to-end run under one protocol. This covers EoRA (SparseGPT 2:4 + GPTQ 4-bit + correction), SLiM, HASSLE-free and 3BASiL.

Q3. Task-specific calibration and a single checkpoint. In the submission, SparseGPT used WikiText2 for every task, but the corrections inside the loop used task data. So for T ≥ 2 the support did depend on the task. We now report two task-independent settings.

  • WikiText2 only: the gain from iterating remains. Joint-SLR from 1 to 5 rounds goes from 11.02 to 10.11 PPL, and from 33.87 to 35.67 ARC-C.
  • One mixed calibration set (WikiText2, ARC, MathQA and GSM8K; 128 sequences in total): a single checkpoint serves all tasks.
Methodb/wPPLARC-CMathQAGSM8K
EoRA, 2:4 + 4-bit3.7813.3233.2829.6811.52
QUENCH3.2912.6133.6231.1211.14
QUENCH (stage 1: 3BASiL), 4-bit3.829.1739.4232.8014.48

On the hardware used for the submission, the shared QUENCH checkpoint reaches 35.32 ARC-C and 8.87 GSM8K. The per-task models reported in the paper reach 34.22 and 7.73.

Thank you for the detailed response with additional experiments. The matched comparisons, ablation, and revised storage accounting address most of my experimental concerns.

However, the new results also confirm that joint-SLR does not improve over prior methods. With the contribution now focusing on quantization scheduling, I find the observation useful but novelty still limited. The practical gain over dense quantization using similar storage remains unclear.

I will maintain my score.

Comments on the paper

1 comment

Thank you to all three reviewers. The reviews agree on three points:

  • the components of QUENCH overlap with HASSLE-free, 3BASiL and LeanQuant;
  • the baselines were not run end to end under one protocol;
  • the result on when to quantise is the most useful part of the paper.

We have revised the paper around these points.

Revised contribution. We no longer present Joint-SLR or EBQ as new algorithms. Joint-SLR is an instance of the HASSLE-free/3BASiL family, and EBQ is LeanQuant without weighting or error feedback. The contribution is a recipe for adding low-bit quantisation to any iterative sparse + low-rank method. It rests on two findings:

  1. Quantise once, after the sparse support has settled. The low-rank correction changes which weights survive pruning, and quantising inside the loop interferes with this. HASSLE-free, 3BASiL and OATS do not quantise, and SLiM quantises first.
  2. After the final correction, the quantiser does not need activation weighting. Plain per-row k-means is sufficient.

New experiments. All numbers are from our own runs. Within each table, every row uses the same calibration data, correction rank and evaluation code.

  • Unless stated otherwise: LLaMA-3-8B, 2:4 sparsity, rank 128, 3-bit, WikiText2 calibration.
  • Storage (b/w) counts the 2:4 metadata, the codebooks and the low-rank factors.

Table G1. Quantise last vs quantise in the loop (QUENCH, 5 rounds, mean ± s.d. over 3 calibration seeds).

SchedulePPL ↓ARC-C ↑MathQA ↑
Quantise last12.09 ± 0.1331.48 ± 0.5326.83 ± 0.45
Quantise in the loop16.74 ± 0.1927.39 ± 0.7425.70 ± 0.48

The ordering holds when stage 1 is replaced.

  • HASSLE-free: quantising last gives lower perplexity at 5 iterations (11.99 vs 12.27) and at 80 iterations (11.31 vs 11.44).
  • 3BASiL: quantising inside its ADMM loop diverges, while quantising last is stable (10.06).

Table G2. End-to-end comparison at matched rank and calibration.

Methodb/wPPL ↓ARC-C ↑
Dense (16-bit)166.1450.51
EoRA, 2:4 + 4-bit3.7812.5428.50
QUENCH (stage 1: Joint-SLR)3.2912.2431.31
QUENCH (stage 1: HASSLE-free, 80 it.)3.2911.3130.72
QUENCH (stage 1: 3BASiL)3.2910.0633.19
QUENCH (stage 1: 3BASiL), 4-bit3.828.7337.80

Every stage 1 improves on EoRA 4-bit while using 0.49 fewer bits per weight. At EoRA's storage, the 4-bit variant lowers perplexity by 3.8 and raises ARC-C by 9.3 points.

Further results, detailed in the individual replies:

  • Mechanism (F2vq): fixing the sparse mask after round 1 removes 56–64% of the gain from iterating.
  • Quantiser (ivHt): with the correction, the 15.1-point perplexity gap between unweighted EBQ and LeanQuant falls to 0.1.
  • Calibration (F2vq): a single checkpoint, calibrated once on a mixture of four datasets, serves all tasks.
  • Qwen3-8B (ivHt, 4jCk): QUENCH with 3BASiL as stage 1 beats EoRA 4-bit on all four benchmarks with 0.48 fewer bits per weight.
  • Also added: SLiM at matched bits and rank; Wanda and magnitude pruning; 2- to 4-bit; compression time, memory, latency and throughput.

Changes to the paper.

  • The abstract's comparison against EoRA's published numbers is replaced by our matched runs: 12.24 / 31.31 vs 12.54 / 28.50, with 0.49 fewer bits per weight.
  • All baseline rows in the main comparisons are now our own end-to-end runs.
  • Storage includes the 2:4 metadata.
  • The new results appear in Section 5 and Appendix D; the per-task results of the submission are in Appendix E.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.