- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces QSLR, which decomposes LLM weights into a quantized matrix, a sparse correction, and a low-rank correction to compress the main weights, handle outliers, and capture shared structure. It uses ADMM to jointly optimize all three components, minimizing layer-output reconstruction error while allowing the components to adapt to one another. Experiments show that joint optimization and mixed corrections generally preserve model quality better under the same storage budget, while custom GPU kernels demonstrate potential inference speedups.
Critical strengths
- The joint formulation is clear and allows quantized, sparse, and low-rank components to adapt to one another. The paper also derives an exact joint projection for fixed scalar quantization grids.
- Comparisons between joint and sequential fitting, and between mixed and single corrections, help separate the benefits of the optimization procedure from those of the representation.
Critical weaknesses
- The need for the quantized component is not sufficiently established. The paper compares , , and , but lacks an baseline under the same storage budget.
- The source of the speedup and the kernel design are insufficiently explained. The dense computation remains, with additional sparse and low-rank corrections. Speedups may come from reduced memory traffic, but the paper provides little detail on dequantization, unstructured sparse computation, or fusion of the three branches.
- The inference benchmarks use only four Transformer blocks and a fixed batch size of eight. Some matrix shapes are slower than FP16, so the practical gains for full models remain unclear.
Questions for authors
- At the same storage budget, how much does improve over ? Could increasing the rank or the number of nonzero entries achieve similar quality without the quantized component?
- Where does the speedup come from? How are quantized weights dequantized and sparse corrections computed, and are the three branches fused? Could the authors provide a breakdown of runtime and memory traffic to show whether the savings outweigh the additional computation?
- Could the authors benchmark full models across different batch sizes and context lengths, comparing the speed and memory usage of , , , and at comparable model quality?
2 comments
W3:
We thank the reviewer for the suggestion. Below, we report full-model results. Our original RTX 4090 is no longer available to us, so all new measurements are on an RTX A6000; we re-ran the 4-block setting on the same GPU so that the two are directly comparable.
Full Llama-7B (all 32 blocks), 3-bit Q + 1% S + rank-16 L, one decode step (auto kernel selection: CUDA-core kernel at batch 1, tensor-core kernel at batch 8).
| Batch | Context | FP16 (tokens/s) | QSLR (tokens/s) | Speedup |
|---|---|---|---|---|
| 1 | 128 | 44 | 93 | 2.08x |
| 1 | 2048 | 39 | 72 | 1.86x |
| 8 | 128 | 342 | 492 | 1.44x |
| 8 | 2048 | 214 | 268 | 1.25x |
Total weight memory (including the FP16 embeddings and LM head) drops from 12.55 GB to 3.83 GB. With the same setup, the speedups of the first 4 blocks and of the full model compare as follows:
| Batch | Context | 4 blocks | 32 blocks (full) |
|---|---|---|---|
| 1 | 128 | 1.87x | 2.08x |
| 1 | 2048 | 1.72x | 1.86x |
| 8 | 128 | 1.39x | 1.44x |
| 8 | 2048 | 1.24x | 1.25x |
The 4-block speedups in fact slightly underestimate the full-model ones. This is because the FP16 embeddings and LM head add a fixed cost to every step, which is a larger share of a 4-block model than of the full model. For a discussion on larger batch sizes, see the next point.
Q3:
Below, we extend the full Llama-7B results (all 32 blocks) on an RTX A6000 to more batch sizes and context lengths, for , , , and .
Configuration. is 3-bit, as in the paper. uses 1% outliers and rank 16. We use 1.3% outliers for and rank 64 for , so that all three match in average storage in the transformer blocks (about 4.4 bits per parameter, bpp). For , we use 7% outliers and rank 352 (also about 4.4 bpp). Since our kernels require a quantized component, we run using cuSPARSE and cuBLAS kernels. In all cases, embeddings and the LM head stay in FP16, and kernels are selected automatically per layer shape.
Speedup over FP16 (one decode step; context is the number of tokens in the KV cache):
| Batch | Context | FP16 (tokens/s) | ||||
|---|---|---|---|---|---|---|
| 1 | 128 | 44 | 2.08x | 2.20x | 1.99x | 1.75x |
| 1 | 4096 | 35 | 1.70x | 1.76x | 1.65x | 1.51x |
| 8 | 128 | 342 | 1.44x | 1.45x | 1.46x | 0.36x |
| 8 | 4096 | 153 | 1.17x | 1.17x | 1.17x | 0.22x |
| 16 | 128 | 627 | 0.98x | 0.99x | 0.99x | 0.29x |
| 16 | 4096 | 206 | 1.01x | 1.01x | 1.01x | 0.18x |
| 32 | 128 | 1115 | 0.63x | 0.64x | 0.65x | 0.13x |
| 32 | 4096 | OOM | OOM | OOM | OOM | OOM |
Memory. Total weight memory (including the FP16 embeddings and LM head) drops from 12.55 GB for FP16 to 3.8 GB for the three -based configurations and 4.7 GB for (cuSPARSE stores 6 bytes per outlier rather than the 4 bytes of our format). At batch 16 and context 4096, peak memory is 44.7 GB for FP16 (close to the A6000's 48 GB), 36.0-36.2 GB for the -based configurations, and 39.8 GB for . At batch 32 and context 4096, all configurations run out of memory, as the KV cache alone needs 64 GB.
Batch size. Our kernels target small-batch decoding, where each step reads all the weights to process only a few tokens, so compressing the weights translates directly into speed due to less data movement. In this regime, QSLR is 1.2-2.1x faster than FP16. As the batch size grows, the FP16 baseline's cost stays nearly constant (22.5 ms per step at batch 1 vs. 28.7 ms at batch 32, context 128), since it remains memory-bound. Our kernel's cost, in contrast, grows with the batch size. Therefore, while our kernels are faster than the FP16 baseline at batch sizes 1 and 8, the speedup vanishes at batch 16, and QSLR is slower than FP16 at batch 32.
We thank the reviewer for the constructive review, and for noting that our joint formulation lets the three components adapt to one another and that our comparisons separate the optimization and representation effects.
W1 & Q1:
Following the reviewer's suggestion, we compare QSLR against an S+L baseline under the same storage budget, using the state-of-the-art S+L solver 3BASiL:
Table Rebuttal 4. Q+S+L versus S+L at the same BPW storage.
At the same storage budget, removing the quantized component dramatically increases the WT-2 perplexity from 17.04 to 184.76 (unusable model) on Llama-3.2-1B, even with the specialized 3BASiL solver. The same holds for Qwen3.8-27B. We anticipated these results that's why S+L was not posed as a real competitor in our experiments. Following the reviewer's suggestion, we will add a similar discussion in the Appendix of the revised version.
W2 & Q2:
We thank the reviewer for their insightful comment. Below, we summarize how the kernels work and where the speedup comes from, and we will add a dedicated section with full implementation details in the next revision.
Source of the speedup. Decoding is memory-bound: most of the time goes into reading the weights from GPU memory, while the computation per weight is small. QSLR stores Q at low bits per weight, a very sparse S, and L and R in FP16. A layer therefore moves significantly fewer bytes than its FP16 counterpart (see the breakdown below).
Kernel design. The kernel uses two parts. The first part computes (Q+S)x. Q and S stay compressed in GPU memory and are decompressed on the fly, right before they are multiplied, so the FP16 matrix is never materialized. Dequantization follows SpQR's format [1]: every group of 16 weights has a 3-bit scale and zero-point, which are themselves quantized with FP16 statistics shared by a 16x16 tile; the kernel unpacks these in registers, uses them in FP16 arithmetic, and accumulates products in FP32. We provide two variants. The CUDA-core variant builds on SpQR's kernel and computes (Q+S)x in a single pass. The tensor-core variant reorders (i.e., swizzles) the quantized weights once at load time, so that the reordered values feed the GPU's tensor cores directly. Sx is then handled by a separate kernel that processes one row per warp and reads each outlier once for all tokens in the batch.
Second, LRx is computed as L(Rx): a short first pass computes the small vector Rx, and the multiplication by L is fused into the main kernel and added just before the output is written. In summary, L is always fused into the main kernel, and S is fused in the CUDA-core variant and separate in the tensor-core variant, and Rx requires its own pass, since it reduces over the entire input. For each layer shape, we pick the faster of the two variants (e.g., on an RTX A6000, the CUDA-core variant at batch 1 and the tensor-core variant at batch 8).
Breakdown. Below, we report the memory traffic and per-kernel GPU time of one layer (RTX A6000, batch 8, 1% outliers, rank 16, tensor-core variant, times from the PyTorch profiler).
The main kernel (Qx together with the fused L(Rx)) dominates the runtime. Sx accounts for 12-18% of it. The savings in memory traffic therefore clearly outweigh the additional computation, even though the speedup stays below the 3.5x traffic reduction, as expected.
[1] SpQR: https://arxiv.org/pdf/2306.03078