acceptodds
Under review as a conference paper at ICLR 2027

QSLR: A Quantized + Sparse + Low-Rank Decomposition of LLM Weights

Abstract

Post-training Quantization (PTQ) of Large Language Models (LLMs) is a popular approach to reduce inference costs but can be restrictive under aggressive compression constraints. Hence, recent methods augment quantized weights with (i) sparse corrections to capture outliers (e.g., SpQR is a specialized solver for the quantized-plus-sparse format) or (ii) low-rank corrections to capture shared structure. We combine both in a unified quantized-plus-sparse-plus-low-rank representation, ,and propose an ADMM-based algorithm that fits all three components jointly. Our novel ADMM framework builds upon computationally efficient subroutines and is agnostic to the quantization format. Extensive experiments demonstrate our joint optimization improves upon sequential fitting procedures. reduces the WikiText-2 perplexity of a Llama-3.2-1B from to compared to fitting SpQR followed by a rank-16 correction, at the same compression budget. Additionally, we show that splitting the allocation between sparsity and low rank can improve performance over using either correction alone. For Qwen3-4B with (sym 2-bit) quantization at 2.47 bits per weight, using to fit all three allocations, the mixed representation achieves a WikiText-2 perplexity of 9.76, reducing the perplexity gap to the dense model by 30.9% relative to and 35.2% relative to . Finally, we design GPU kernels that support and establish speedups over FP16 dense computation in matrix-multiplication and decoding benchmarks.

Primary area:
optimization
Keywords:
Quantization, Pruning, Low-Rank, Efficient ML, Model Compression, Non-Uniform Compression, ADMM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CcRROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces QSLR, which decomposes LLM weights into a quantized matrix, a sparse correction, and a low-rank correction to compress the main weights, handle outliers, and capture shared structure. It uses ADMM to jointly optimize all three components, minimizing layer-output reconstruction error while allowing the components to adapt to one another. Experiments show that joint optimization and mixed corrections generally preserve model quality better under the same storage budget, while custom GPU kernels demonstrate potential inference speedups.

Critical strengths
  1. The joint formulation is clear and allows quantized, sparse, and low-rank components to adapt to one another. The paper also derives an exact joint projection for fixed scalar quantization grids.
  2. Comparisons between joint and sequential fitting, and between mixed and single corrections, help separate the benefits of the optimization procedure from those of the representation.
Critical weaknesses
  1. The need for the quantized component is not sufficiently established. The paper compares Q+S+LQ+S+L, Q+SQ+S, and Q+LQ+L, but lacks an S+LS+L baseline under the same storage budget.
  2. The source of the speedup and the kernel design are insufficiently explained. The dense XQXQ computation remains, with additional sparse and low-rank corrections. Speedups may come from reduced memory traffic, but the paper provides little detail on dequantization, unstructured sparse computation, or fusion of the three branches.
  3. The inference benchmarks use only four Transformer blocks and a fixed batch size of eight. Some matrix shapes are slower than FP16, so the practical gains for full models remain unclear.
Questions for authors
  1. At the same storage budget, how much does Q+S+LQ+S+L improve over S+LS+L? Could increasing the rank or the number of nonzero entries achieve similar quality without the quantized component?
  2. Where does the speedup come from? How are quantized weights dequantized and sparse corrections computed, and are the three branches fused? Could the authors provide a breakdown of runtime and memory traffic to show whether the savings outweigh the additional computation?
  3. Could the authors benchmark full models across different batch sizes and context lengths, comparing the speed and memory usage of Q+S+LQ+S+L, Q+SQ+S, Q+LQ+L, and S+LS+L at comparable model quality?
2 comments

We thank the reviewer for the constructive review, and for noting that our joint formulation lets the three components adapt to one another and that our comparisons separate the optimization and representation effects.

W1 & Q1:

Following the reviewer's suggestion, we compare QSLR against an S+L\mathbf{S}+\mathbf{L} baseline under the same storage budget, using the state-of-the-art S+L\mathbf{S}+\mathbf{L} solver 3BASiL:

ModelMethodBPWWT2C40-shot (%)MMLU (%)
Llama-3.2-1BS+L\mathbf{S}+\mathbf{L}, 3BASiL3.14184.76793.4834.8125.42
Q+S+L\mathbf{Q}+\mathbf{S}+\mathbf{L}, QSLR3.1317.0434.6544.6925.64
Qwen3.8-27BS+L\mathbf{S}+\mathbf{L}, 3BASiL3.0240.84159.7235.1425.25
Q+S+L\mathbf{Q}+\mathbf{S}+\mathbf{L}, QSLR3.028.0313.6066.5374.43

Table Rebuttal 4. Q+S+L\mathbf{Q}+\mathbf{S}+\mathbf{L} versus S+L\mathbf{S}+\mathbf{L} at the same BPW storage.

At the same storage budget, removing the quantized component dramatically increases the WT-2 perplexity from 17.04 to 184.76 (unusable model) on Llama-3.2-1B, even with the specialized 3BASiL solver. The same holds for Qwen3.8-27B. We anticipated these results that's why S+L\mathbf{S}+\mathbf{L} was not posed as a real competitor in our experiments. Following the reviewer's suggestion, we will add a similar discussion in the Appendix of the revised version.

W2 & Q2:

We thank the reviewer for their insightful comment. Below, we summarize how the kernels work and where the speedup comes from, and we will add a dedicated section with full implementation details in the next revision.

Source of the speedup. Decoding is memory-bound: most of the time goes into reading the weights from GPU memory, while the computation per weight is small. QSLR stores QQ at low bits per weight, a very sparse SS, and LL and RR in FP16. A layer therefore moves significantly fewer bytes than its FP16 counterpart (see the breakdown below).

Kernel design. The kernel uses two parts. The first part computes (Q+S)x(Q + S) x. QQ and SS stay compressed in GPU memory and are decompressed on the fly, right before they are multiplied, so the FP16 matrix is never materialized. Dequantization follows SpQR's format [1]: every group of 16 weights has a 3-bit scale and zero-point, which are themselves quantized with FP16 statistics shared by a 16x16 tile; the kernel unpacks these in registers, uses them in FP16 arithmetic, and accumulates products in FP32. We provide two variants. The CUDA-core variant builds on SpQR's kernel and computes (Q+S)x(Q + S) x in a single pass. The tensor-core variant reorders (i.e., swizzles) the quantized weights once at load time, so that the reordered values feed the GPU's tensor cores directly. SxS x is then handled by a separate kernel that processes one row per warp and reads each outlier once for all tokens in the batch.

Second, LRxLR x is computed as L(Rx)L(Rx): a short first pass computes the small vector RxRx, and the multiplication by LL is fused into the main kernel and added just before the output is written. In summary, LL is always fused into the main kernel, and SS is fused in the CUDA-core variant and separate in the tensor-core variant, and RxRx requires its own pass, since it reduces over the entire input. For each layer shape, we pick the faster of the two variants (e.g., on an RTX A6000, the CUDA-core variant at batch 1 and the tensor-core variant at batch 8).

Breakdown. Below, we report the memory traffic and per-kernel GPU time of one layer (RTX A6000, batch 8, 1% outliers, rank 16, tensor-core variant, times from the PyTorch profiler).

Layer (m x n)FP16 traffic (MB)QSLR traffic (MB)FP16 (µs)RxRx pass (µs)SxSx (µs)Main kernel (µs)QSLR total (µs)
4096 x 409633.79.751.34.07.027.538.6
4096 x 1100890.425.6131.65.611.267.184.0
11008 x 409690.425.6144.74.212.155.872.2
16384 x 16384537.4147.6788.65.944.3317.3367.4

The main kernel (QxQ x together with the fused L(Rx)L(Rx)) dominates the runtime. SxS x accounts for 12-18% of it. The savings in memory traffic therefore clearly outweigh the additional computation, even though the speedup stays below the 3.5x traffic reduction, as expected.

[1] SpQR: https://arxiv.org/pdf/2306.03078

W3:

We thank the reviewer for the suggestion. Below, we report full-model results. Our original RTX 4090 is no longer available to us, so all new measurements are on an RTX A6000; we re-ran the 4-block setting on the same GPU so that the two are directly comparable.

Full Llama-7B (all 32 blocks), 3-bit Q + 1% S + rank-16 L, one decode step (auto kernel selection: CUDA-core kernel at batch 1, tensor-core kernel at batch 8).

BatchContextFP16 (tokens/s)QSLR (tokens/s)Speedup
112844932.08x
1204839721.86x
81283424921.44x
820482142681.25x

Total weight memory (including the FP16 embeddings and LM head) drops from 12.55 GB to 3.83 GB. With the same setup, the speedups of the first 4 blocks and of the full model compare as follows:

BatchContext4 blocks32 blocks (full)
11281.87x2.08x
120481.72x1.86x
81281.39x1.44x
820481.24x1.25x

The 4-block speedups in fact slightly underestimate the full-model ones. This is because the FP16 embeddings and LM head add a fixed cost to every step, which is a larger share of a 4-block model than of the full model. For a discussion on larger batch sizes, see the next point.

Q3:

Below, we extend the full Llama-7B results (all 32 blocks) on an RTX A6000 to more batch sizes and context lengths, for Q+S+LQ+S+L, Q+SQ+S, Q+LQ+L, and S+LS+L.

Configuration. QQ is 3-bit, as in the paper. Q+S+LQ+S+L uses 1% outliers and rank 16. We use 1.3% outliers for Q+SQ+S and rank 64 for Q+LQ+L, so that all three match in average storage in the transformer blocks (about 4.4 bits per parameter, bpp). For S+LS+L, we use 7% outliers and rank 352 (also about 4.4 bpp). Since our kernels require a quantized component, we run S+LS+L using cuSPARSE and cuBLAS kernels. In all cases, embeddings and the LM head stay in FP16, and kernels are selected automatically per layer shape.

Speedup over FP16 (one decode step; context is the number of tokens in the KV cache):

BatchContextFP16 (tokens/s)Q+S+LQ+S+LQ+SQ+SQ+LQ+LS+LS+L
1128442.08x2.20x1.99x1.75x
14096351.70x1.76x1.65x1.51x
81283421.44x1.45x1.46x0.36x
840961531.17x1.17x1.17x0.22x
161286270.98x0.99x0.99x0.29x
1640962061.01x1.01x1.01x0.18x
3212811150.63x0.64x0.65x0.13x
324096OOMOOMOOMOOMOOM

Memory. Total weight memory (including the FP16 embeddings and LM head) drops from 12.55 GB for FP16 to 3.8 GB for the three QQ-based configurations and 4.7 GB for S+LS+L (cuSPARSE stores 6 bytes per outlier rather than the 4 bytes of our format). At batch 16 and context 4096, peak memory is 44.7 GB for FP16 (close to the A6000's 48 GB), 36.0-36.2 GB for the QQ-based configurations, and 39.8 GB for S+LS+L. At batch 32 and context 4096, all configurations run out of memory, as the KV cache alone needs 64 GB.

Batch size. Our kernels target small-batch decoding, where each step reads all the weights to process only a few tokens, so compressing the weights translates directly into speed due to less data movement. In this regime, QSLR is 1.2-2.1x faster than FP16. As the batch size grows, the FP16 baseline's cost stays nearly constant (22.5 ms per step at batch 1 vs. 28.7 ms at batch 32, context 128), since it remains memory-bound. Our kernel's cost, in contrast, grows with the batch size. Therefore, while our kernels are faster than the FP16 baseline at batch sizes 1 and 8, the speedup vanishes at batch 16, and QSLR is slower than FP16 at batch 32.

Reviewer_LtZxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training compression of large language models. It decomposes model weights into a sum of quantized, sparse, and low-rank components. Sparse and low-rank corrections capture complementary residual structures. The authors argue that joint updates are more effective than sequential fitting that freezes previously fitted components. The proposed QSLR algorithm uses ADMM to minimize an activation-aware reconstruction error. It provides a joint projection for the quantized and sparse components under fixed scalar quantization grids. The experiments examine joint versus sequential fitting, correction allocation under a fixed storage budget, and GPU acceleration. The results show that joint fitting and mixed correction allocation reduce perplexity.

Critical strengths
  1. The mixed representation has a clear motivation. The experiments under a fixed storage budget provide important supporting evidence. The authors use QSLR for sparse-only correction, low-rank-only correction, and mixed allocation. This helps distinguish the benefits of the representation from those of solver improvements.
  2. The comparison between joint and sequential fitting helps explain why QSLR improves performance. These controlled experiments provide credible evidence. Together with the fixed-budget experiments, they support the paper’s main conclusions on compression.
Critical weaknesses
  1. The experiments support the benefits of jointly updating the components. However, the specific contribution of ADMM is not sufficiently explained or validated.
  2. The current experiments do not fully support the claim of end-to-end inference speedup. The reported speedups demonstrate gains under the tested configuration. However, they do not fully establish the decoding performance of the complete Llama-30B model.
Questions for authors
  1. In Table 1, do QSLR and LR+QS use the same initial low-rank component? Are their quantization grids numerically identical? These details affect the assessment of the algorithm’s contribution.
  2. How do the decoding measurements on four Transformer blocks translate to full-model performance under the same compression configuration? This could affect the paper’s main conclusions.
3 comments

We thank the reviewer for the constructive review, and for highlighting the clear motivation of the mixed representation and that our fixed-budget and joint-versus-sequential experiments provide credible evidence for the paper's main conclusions. We address the remaining concerns below.

W1:

We thank the reviewer for acknowledging that our experiments support the benefits of jointly updating the components. Below, we explain why ADMM is what makes this joint update possible, and validate its specific contribution against alternatives that either fit the same components sequentially or update them jointly without ADMM.

Explanation. The layerwise loss 12∥H1/2(W−Q−S−L)∥F2\tfrac12\Vert{\bf H}^{1/2}({\bf W-Q-S-L})\Vert_F^2 couples the decomposition of the weights through the Hessian H\bf H, while the constraints are combinatorial (a fixed quantization grid, kk nonzeros, rank rr). Projecting onto the quantized-plus-sparse set (or even the quantized-only or sparse-only set) in the H\bf H-weighted norm has no closed-form solution. This is why GPTQ and SparseGPT rely on column-by-column error feedback, and why alternating schemes such as OATS replace H\bf H by its diagonal, under which each component has a cheap closed-form update. With the full H\bf H, ADMM separates the two difficulties: a smooth copy V\bf V handles the H\bf H-coupling exactly (one linear solve that reuses a single eigendecomposition of H\bf H), the (Q,S)({\bf Q,S})-update becomes an unweighted projection that we compute exactly (Proposition 3.1), and L\bf L is a weighted rank-rr fit. The dual variable keeps the three components adapting to each other, and Theorem 3.2 guarantees convergence. In short, ADMM is a joint optimization procedure whose iterations consist of highly efficient subroutines, all while minimizing the layerwise reconstruction error with the full Hessian.

Validation against simple baselines. We validate the specific contribution of ADMM in two ways. First, the paper's QS_ADMM+LR and LR+QS_ADMM baselines use exactly our own subroutines but fit them in separate stages; they answer the question do we need a truly joint ADMM framework, or is a Q+S\bf Q+S solver plus a low-rank correction good enough? On Llama-3.2-1B (sym W2, 1%, rank 16), ADMM reaches 15.41 WT2 perplexity versus 18.98 for the better of the two orders (Table 1). Second, to separate ADMM from joint updating itself, we design an alternative that also updates the three components jointly but without ADMM: we extend OATS to Q+S+L\bf Q+S+L, starting from a weighted rank-16 fit and alternating Q←RTNh(W−S−L){\bf Q}\leftarrow\mathrm{RTN_h}({\bf W-S-L}), S←TopKh(W−Q−L){\bf S}\leftarrow\mathrm{TopK_h}({\bf W-Q-L}) and L←rank-16h(W−Q−S){\bf L}\leftarrow\mathrm{rank\text{-}16_h}({\bf W-Q-S}) in OATS's "outlier-aware" space h=diag(H)h=\mathrm{diag}({\bf H}), for 80 alternating minimization steps (the default value in OATS). This is the natural extension of OATS to our additive constraints.

We also apply the low-rank recovery methods QERA, GlowQ and SRR proposed by reviewer 5Bje on top of our own Q+S\bf Q+S solver QS_ADMM.

Llama-3.2-1B (2.5 BPW)Qwen3.8-27B (2.4 BPW)
MethodWT2C40-shotMMLUWT2C40-shotMMLU
QS_ADMM→QERA18.8542.0643.8425.108.3714.3466.6372.54
QS_ADMM→GlowQ19.3543.2843.6924.378.3714.3566.7774.14
QS_ADMM→SRR19.1341.3944.2125.648.4214.3265.6671.27
OATS-style alternating minimization39.4259.1842.1025.648.4313.6567.6077.45
QSLR15.3328.9546.1026.157.9313.1168.3778.64

Table Rebuttal 3. Our Q+S\bf Q+S solver augmented with low-rank fitting and OATS-style alternating minimization versus QSLR's ADMM for Q+S+L\bf Q+S+L. Symmetric W2, 1% sparsity, rank 16. 0-shot is the mean accuracy over PIQA, ARC-e, ARC-c, HellaSwag, WinoGrande, OpenBookQA, BoolQ and RTE. MMLU is reported using 5-shot.

ADMM outperforms both alternatives on every metric for both models. The QS_ADMM→correction rows use our own Q+S\bf Q+S solver followed by a state-of-the-art correction, yet joint fitting still reduces WT2 perplexity from 18.85 to 15.33 on Llama-3.2-1B and improves MMLU on Qwen3.8-27B from 74.14 to 78.64.

The OATS-style baseline also employs an alternating minimization procedure, but relies on a diagonal approximation of H\bf H; in the harder SpQR(2,64,3,16) format on Qwen3.8-27B (2.55 BPW), the gap to this baseline widens to 11.8 MMLU points (63.45 to 75.27).

Since both procedures update the components jointly, this isolates the specific contribution of ADMM: handling the full Hessian exactly gives better downstream performance than a joint update with a diagonal approximation of H\bf H.

W2 & Q2:

We thank the reviewer for the suggestion. Below, we report full-model results. Our original RTX 4090 is no longer available to us, so all new measurements are on an RTX A6000; we re-ran the 4-block setting on the same GPU so that the two are directly comparable.

We first note that, while the full Llama-30B fits on a single 48 GB A6000 in our QSLR format, its FP16 baseline (about 65 GB of weights) does not. Additionally, running the baseline across multiple GPUs would skew the comparison in favor of QSLR, since the FP16 baseline would suffer from its heavier GPU communications compared to QSLR, which does not reflect efficiency of our kernels. We therefore evaluate the full Llama-7B instead, whose FP16 baseline fits on one GPU.

Full Llama-7B (all 32 blocks), 3-bit Q + 1% S + rank-16 L, one decode step (auto kernel selection: CUDA-core kernel at batch 1, tensor-core kernel at batch 8):

BatchContextFP16 (tokens/s)QSLR (tokens/s)Speedup
112844932.08x
1204839721.86x
81283424921.44x
820482142681.25x

Total weight memory (including the FP16 embeddings and LM head) drops from 12.55 GB to 3.83 GB. With the same setup, the speedups of the first 4 blocks and of the full model compare as follows:

BatchContext4 blocks32 blocks (full)
11281.87x2.08x
120481.72x1.86x
81281.39x1.44x
820481.24x1.25x

The 4-block speedups in fact slightly underestimate the full-model ones. This is because the FP16 embeddings and LM head add a fixed cost to every step, which is a larger share of a 4-block model than of the full model. Since Llama-30B consists of identical blocks in the same way, we expect its 4-block speedups to likewise be a conservative estimate of full-model performance.

Q1:

The reviewer is exactly right. Both methods use the same initial low-rank component. Additionally, the quantization grids are numerically identical and are fixed until the end of compression for both methods. All downstream differences in performance are entirely due to the joint fitting provided by ADMM. We will make this more explicit in the revised version.

Thank you for the detailed responses and additional experiments. The new theoretical explanations and experimental results further improve the clarity and credibility of the proposed method. Considering both the paper's novelty and its remaining limitations, I maintain my original rating of Weak Acceptance.

Reviewer_5BjeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a QSLR method that combines quantization, sparse correction, and low-rank reconstruction to recover quantization errors in LLMs. They introduce an ADMM-based co-optimization framework along with GPU kernels, improving PPL over sequential fitting in various low-bit quantization environments.

Critical strengths
  1. The proposed method integrates different optimization approaches explicitly considering the activation while allocating a fixed budget into sparse and low-rank corrections.
  2. They report that QSLR achieves improvements in PPL compared to its alternatives through the ablation study.
Critical weaknesses
  1. The paper combines three different approaches, which have been widely studied individually as well as more comprehensively e.g., OATS HASSLE-FREE, and thus the main focus should be on the use of ADMM. However, the evaluation does not sufficiently show the benefits of the specific ADMM choice.
  2. The evaluation mainly reported PPL and KL divergence; the zero-shot accuracy is only evaluated in their ablation study. It also lacks the comparisons to other state-of-the-art baselines. Also, they showed their effectiveness with the improvement in the KL divergence; it was not clearly shown if it would lead to general performance gain as the resulting PPL does not follow the trend.
  3. The inference time evaluation is limited to four transformer blocks rather than showing the end-to-end model speed, e.g., throughput/TTFT.
Questions for authors
  1. In Table 8, as compared to SpQR, QSLR with Q+S improves PPL from 12.65 to 11.43, while the best Q+S+L improves from 10.88 to 10.47. How much improvements were contributed from the mixed representation or other optimization procedures proposed?
  2. Can the authors compare the proposed method with other state-of-the-art baselines? They are many papers that applies the low-rank recovery terms along with the quantization, including QERA [1], SERQ [2], and GlowQ [3]. Please also compare the results with commonly-used, various metrics/benchmarks beyond PPL, including zero-shot accuracy and MMLU.
  3. Can the authors clarify why the KL divergence was used the main metric during the evaluation? The trend of KL does not follow the one seen in PPL.
  4. Since the authors claims the kernel-level optimization as one of their contributions, it should provide more details about the related implementation.
  5. Can the authors provide more evidence why the ADMM is a suitable choice to address the issues in integrating the three different mechanisms? For example, it should (at least) have the comparison to simpler hand-crafted alternatives.

[1] Zhang, Cheng, et al. "Qera: an analytical framework for quantization error reconstruction." International Conference on Learning Representations. Vol. 2025. 2025. [2] Park, Yeonsik, Hyeonseong Kim, and Seungkyu Choi. "SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization." arXiv preprint arXiv:2603.08185 (2026). [3] An, Selim, Il Suh, and Yeseong Kim. "GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs." International Conference on Learning Representations. Vol. 2026. 2026.

4 comments

We thank the reviewer for the valuable feedback. We address the concerns in what follows.

W1, Q2 & Q5 (1/2):

In our Introduction, we review the mature layerwise-reconstruction solvers developed for each component individually (quantization, sparsity or low rank), as well as the pairwise solvers built in the same framework: SpQR for Q+S\mathbf{Q}+\mathbf{S}, CALDERA/LoftQ for Q+L\mathbf{Q}+\mathbf{L}, and OATS/HASSLE-free, which the reviewer mentions, for S+L\mathbf{S}+\mathbf{L}.

As far as we know, QSLR is the first method that fits all three components jointly. In what follows, we show that:

  • (i) sequential fitting is a natural and strong hand-crafted alternative to ADMM, and fitting our own Q+S\mathbf{Q}+\mathbf{S} ADMM solver followed by a low-rank term is an ablation that isolates the benefit of ADMM and joint optimization.
  • (ii) following the reviewer's insightful recommendation, we design a further hand-crafted alternative, an alternating-minimization baseline that extends OATS to Q+S+L\mathbf{Q}+\mathbf{S}+\mathbf{L}.
  • (iii) we compare against the low-rank recovery methods QERA, SERQ, GlowQ, and SRR, reporting zero-shot accuracy and MMLU alongside perplexity in our main comparison, following the reviewer's suggestions.

ADMM outperforms all of these alternatives.

Sequential fitting is a standard hand-crafted alternative.

Sequential fitting procedures similar to the ones proposed in our paper are used in recent compression works. For instance, SLiM [1] considers a quantized-and-sparse representation (a different constraint from our additive sparse term) augmented with a low-rank correction: it quantizes the weights, prunes the quantized weights with an off-the-shelf one-shot pruner, and finally fits low-rank adapters to the accumulated error.

EoRA [2] fits the same optimal closed-form low-rank compensation we use in our paper to the residual of an already compressed model, and the low-rank recovery methods mentioned by the reviewer (QERA, SERQ, GlowQ) all fit their correction after compression.

Our paper therefore compares against the strongest such procedures (Tables 1, 2 and 4–7): Best sequential, which tries all six orders of GPTQ (quantization), SparseGPT (pruning) and activation-weighted SVD (low-rank), and SpQR+LR / LR+SpQR, which pair SpQR with an EoRA-style low-rank fit. We also introduce QS_ADMM+LR and LR+QS_ADMM as an ablation of ADMM: they answer the question do we need a truly joint ADMM framework, or is a Q+S\mathbf{Q}+\mathbf{S} solver plus a low-rank correction good enough?

On Llama-3.2-1B (sym W2, 1%, rank 16), ADMM reaches 15.41 WT2 perplexity versus 18.98 for the better of the two orders (Table 1). As explained in line 65 of the paper, every sequential order shares the same limitation: when the low-rank term is fitted last, the quantization grid, the quantized values and the sparse support ignore the correction that follows; when it is fitted first, it ignores the quantization and pruning errors that follow. Joint fitting removes this mismatch, and this is exactly where ADMM shines.

Why ADMM is a suitable choice for integrating the three mechanisms.

The layerwise loss 12∥H1/2(W−Q−S−L)∥F2\tfrac12\Vert\mathbf{H}^{1/2}(\mathbf{W}-\mathbf{Q}-\mathbf{S}-\mathbf{L})\Vert_F^2 couples the decomposition of the weights through the Hessian H\mathbf{H}, while the constraints are combinatorial (a fixed quantization grid, kk nonzeros, rank rr). Projecting onto the quantized-plus-sparse set (or even the quantized-only or sparse-only set) in the H\mathbf{H}-weighted norm has no closed-form solution. This is why GPTQ and SparseGPT rely on column-by-column error feedback, and why alternating schemes such as OATS replace H\mathbf{H} by its diagonal, under which each component has a cheap closed-form update. With the full H\mathbf{H}, ADMM separates the two difficulties: a smooth copy V\mathbf{V} handles the H\mathbf{H}-coupling exactly (one linear solve that reuses a single eigendecomposition of H\mathbf{H}), the (Q,S)(\mathbf{Q},\mathbf{S})-update becomes an unweighted projection that we compute exactly (Proposition 3.1), and L\mathbf{L} is a weighted rank-rr fit. The dual variable keeps the three components adapting to each other, and Theorem 3.2 guarantees convergence. In short, ADMM is a joint optimization procedure whose iterations consist of highly efficient subroutines, all while minimizing the layerwise reconstruction error with the full Hessian.

[1] Mozaffari, Yazdanbakhsh and Mehri Dehnavi, SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression. ICML 2025.

[2] Liu et al., EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation. arXiv:2410.21271, 2025.

W1, Q2 & Q5 (2/2):

Comparison with hand-crafted alternatives, with zero-shot accuracy and MMLU. Following the reviewer's suggestion, we extend OATS to Q+S+L\mathbf{Q}+\mathbf{S}+\mathbf{L}: starting from a weighted rank-16 fit, it alternates Q←RTNh(W−S−L)\mathbf{Q}\leftarrow\mathrm{RTN_h}(\mathbf{W}-\mathbf{S}-\mathbf{L}), S←TopKh(W−Q−L)\mathbf{S}\leftarrow\mathrm{TopK_h}(\mathbf{W}-\mathbf{Q}-\mathbf{L}) and L←rank-16h(W−Q−S)\mathbf{L}\leftarrow\mathrm{rank\text{-}16_h}(\mathbf{W}-\mathbf{Q}-\mathbf{S}) in OATS's "outlier-aware" space h=diag(H)h=\mathrm{diag}(\mathbf{H}), for 80 alternating minimization steps (the default value in OATS). This is the natural extension of OATS to our additive constraints.

We also apply the low-rank recovery methods QERA, GlowQ and SRR on top of three Q+S\mathbf{Q}+\mathbf{S} backends (SparseGPT→GPTQ, GPTQ→SparseGPT and our own QS_ADMM), giving nine hand-crafted sequential pipelines. All methods use the same constraints, calibration data and storage:

Llama-3.2-1B (2.5 BPW)Qwen3.8-27B (2.4 BPW)
MethodWT2C40-shotMMLUWT2C40-shotMMLU
SparseGPT→GPTQ→QERA240.53461.3436.7024.4612.2622.1862.9060.97
SparseGPT→GPTQ→GlowQ165.60322.9537.9325.2512.8922.1662.1754.55
SparseGPT→GPTQ→SRR143.59274.8538.8624.5112.5922.1059.2455.31
GPTQ→SparseGPT→QERA44.91111.1240.4923.799.1415.9965.6771.13
GPTQ→SparseGPT→GlowQ37.7696.3440.5124.818.9315.3966.3571.24
GPTQ→SparseGPT→SRR34.1985.0639.9025.099.0115.3366.2772.86
QS_ADMM→QERA18.8542.0643.8425.108.3714.3466.6372.54
QS_ADMM→GlowQ19.3543.2843.6924.378.3714.3566.7774.14
QS_ADMM→SRR19.1341.3944.2125.648.4214.3265.6671.27
OATS-style alternating minimization39.4259.1842.1025.648.4313.6567.6077.45
QSLR15.3328.9546.1026.157.9313.1168.3778.64

Table Rebuttal 1. Hand-crafted sequential and OATS-style alternating minimization alternatives versus QSLR's ADMM for Q+S+L\mathbf{Q}+\mathbf{S}+\mathbf{L}. Symmetric W2, 1% sparsity, rank 16. MMLU is reported using 5-shot.

ADMM outperforms every hand-crafted alternative on every metric for both models. The QS_ADMM→correction rows use our own Q+S\mathbf{Q}+\mathbf{S} solver followed by a low-rank correction different from the one used in our paper, yet joint fitting with ADMM still reduces the WT2 perplexity gap to the dense model by 39% on Llama-3.2-1B (18.85 to 15.33; dense 9.75) and by 33% on Qwen3.8-27B (8.37 to 7.93; dense 7.05), and improves MMLU on Qwen3.8-27B from 74.14 to 78.64. The OATS-style baseline also updates all three components jointly, but relies on a diagonal approximation of H\mathbf{H}. In the harder SpQR(2,64,3,16) format on Qwen3.8-27B (2.55 BPW), the gap to this baseline widens to 11.8 MMLU points (63.45 to 75.27). This supports our design choice: handling the full Hessian exactly through ADMM gives better downstream performance than hand-crafted alternating minimization with a diagonal approximation of H\mathbf{H}.

SERQ. SERQ is a low-rank correction method designed for joint weight-and-activation quantization and stores its correction as 16 selected INT4 channels. Our low-rank constraint is different. That said, notice that our solver accommodates a Q+S\mathbf{Q}+\mathbf{S} decomposition (when the rank is 0), which we denote by QS_ADMM. In what follows, we compare QS_ADMM against SpQR, a specialized Q+S\mathbf{Q}+\mathbf{S} solver, both augmented with SERQ as a low-rank correction term. We evaluate in the activation quantization setting, which is what SERQ has been designed for.

SpQR+SERQQS_ADMM+SERQ
SettingBPWWT2C4WT2C4
Llama-3.2-1B, g=16g=16, A82.9817.6532.4317.0632.26
Llama-3.2-1B, g=16g=16, A42.9826.9357.5326.3456.11
Qwen3.8-27B, g=64g=64, A82.428.9714.588.6814.40
Qwen3.8-27B, g=64g=64, A42.4210.0215.999.5915.72

Table Rebuttal 2. SERQ with 8-bit (A8) and 4-bit (A4) activations in the SpQR(2,g,3,16) grid with 1% sparsity and rank16.

W2 & Q3:

We use KL divergence as an additional metric to perplexity, following recent state-of-the-art quantization work such as YAQA [1], which minimizes and reports the KL divergence to the original model to measure how well it is preserved. KL compares the full next-token distribution of the compressed model with that of the dense model, while perplexity only scores the target token. The two metrics can slightly disagree, as noted by the reviewer.

In our experiments, we do report perplexity on every compression result; see Tables 1, 4–14, 16 and 17 and Figures 2–5 of the paper. Regarding the zero-shot accuracy and the state-of-the-art baselines, Tables Rebuttal 1 and 2 above compare QSLR with QERA, GlowQ, SRR and SERQ, and Table Rebuttal 1 reports zero-shot accuracy on eight tasks and 5-shot MMLU.

QSLR is the best method on every metric, including zero-shot accuracy (46.10 vs. 44.21 for the best alternative on Llama-3.2-1B, 68.37 vs. 67.60 on Qwen3.8-27B) and MMLU (78.64 vs. 77.45 on Qwen3.8-27B), so our improvements do lead to a large performance gain. Following the reviewer's recommendation, we will include these additional evaluations to the revised manuscript.

[1] Tseng, Sun and De Sa, Model-Preserving Adaptive Rounding (YAQA). arXiv:2505.22988, 2025.

W3:

We thank the reviewer for the suggestion. Below, we report full-model results. Our original RTX 4090 is no longer available to us, so all new measurements are on an RTX A6000; we re-ran the 4-block setting on the same GPU so that the two are directly comparable.

Full Llama-7B (all 32 blocks), 3-bit Q + 1% S + rank-16 L, one decode step (auto kernel selection: CUDA-core kernel at batch 1, tensor-core kernel at batch 8):

BatchContextFP16 (tokens/s)QSLR (tokens/s)Speedup
112844932.08x
1204839721.86x
81283424921.44x
820482142681.25x

Total weight memory (including the FP16 embeddings and LM head) drops from 12.55 GB to 3.83 GB. With the same setup, the speedups of the first 4 blocks and of the full model are as follows.

BatchContext4 blocks32 blocks (full)
11281.87x2.08x
120481.72x1.86x
81281.39x1.44x
820481.24x1.25x

The 4-block speedups in fact slightly underestimate full model speedups. This is because the FP16 embeddings and LM head add a fixed cost to every step, which is a larger share of a 4-block model than of the full model.

Regarding TTFT: our kernels target decoding, which is memory-bound, since each step must read all the weights while doing little computation per byte. Compressing the weights therefore translates directly into speed. TTFT, in contrast, is dominated by prefill, which processes the whole prompt at once and is typically compute-bound, so it falls in a different regime. Therefore, consistent with prior work on weight compression, especially involving sparsity, e.g., SpQR [1], we focus on decoding speed.

[1] SpQR: https://arxiv.org/pdf/2306.03078

Q1:

Table 8 separates the two effects at the same 2.81 BPW (Qwen3-4B-Base, SpQR(2,64,3,16)). Optimization: replacing SpQR's solver by ours in the same Q+S\mathbf{Q}+\mathbf{S} representation reduces WT2 perplexity from 12.65 to 11.43, about 56% of the total gain to 10.47. Mixed representation: with our solver fitting every allocation, splitting the budget between sparsity and low rank reduces it further to 10.47 (from 11.43 for Q+S\mathbf{Q}+\mathbf{S} or 10.88 for Q+L\mathbf{Q}+\mathbf{L}), the remaining 44%; sequential SpQR+LR cannot recover this gain even with more storage (12.25 at 3.31 BPW).

Q4:

We thank the reviewer for their comment. Below, we briefly summarize how the kernels work, and we will add a dedicated section with full implementation details in the next revision.

The kernel uses two parts. The first part computes (Q+S)x(Q + S) x. QQ and SS stay compressed in GPU memory and are decompressed on the fly, right before they are multiplied, so the FP16 matrix is never materialized. Dequantization follows SpQR's format [1]: every group of 16 weights has a 3-bit scale and zero-point, which are themselves quantized with FP16 statistics shared by a 16x16 tile; the kernel unpacks these in registers, uses them in FP16 arithmetic, and accumulates products in FP32. We provide two variants. The CUDA-core variant builds on SpQR's kernel and computes (Q+S)x(Q + S) x in a single pass. The tensor-core variant reorders (i.e., swizzles) the quantized weights once at load time, so that the reordered values feed the GPU's tensor cores directly. SxS x is then handled by a separate kernel that processes one row per warp and reads each outlier once for all tokens in the batch.

Second, to compute LRxLR x, it is computed as L(Rx)L(Rx): a short first pass computes the small vector RxRx, and the multiplication by LL is fused into the main kernel and added just before the output is written. As a result, the low-rank component adds only a small overhead per layer.

For each layer shape, we pick the faster of the two variants (e.g., on an RTX A6000, the CUDA-core variant at batch 1 and the tensor-core variant at batch 8).

[1] SpQR: https://arxiv.org/pdf/2306.03078

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.