- Rating
- 4
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoCaQ, a training-free framework that couples low-rank SVD weight compression and low-bit post-training quantization (PTQ) for flexible LLM deployment under tight memory budgets. The authors observe two key issues when combining compression and quantization sequentially: 1) layer-wise rank structures that are optimal before quantization do not maintain their relative ranking after low-bit quantization, and 2) decomposing a weight matrix W into low-rank factors A_r and B_r introduces an intermediate activation H = B_r X with severe outliers that degrade low-bit activation quantization. To solve these issues, CoCaQ introduces Quantization-aware Model Compression (QaMC)—combining Quantization-Friendly SVD (QF-SVD) with Quantization-aware Layer-wise Compression Allocation (QLCA) based on post-quantization output KL divergence—and Activation Stabilization for Quantization (ASQ), which inserts an orthogonal rotation matrix R* between the low-rank factors to smooth intermediate activation outliers. Experiments on LLaMA-2, LLaMA-3, and Qwen3 models demonstrate improved accuracy-memory trade-offs over standalone compression, standalone quantization, and sequential or joint compression-plus-quantization baselines.
Critical strengths
-
The paper clearly identifies and empirically illustrates (Figure 2) two concrete bottlenecks in combining low-rank compression with quantization: the rank-order shift of layer-wise structures after quantization and the emergence of activation outliers in the intermediate low-rank bottleneck H = B_r X.
-
The proposed training-free solutions—Frobenius norm balancing in QF-SVD (Equation 8), orthogonal rotation smoothing across the low-rank factors in ASQ (Equation 12), and post-quantization KL rank search in QLCA—are mathematically clean and achieve strong gains over SVD-LLM, KFAC-SVD, and UniQL across five LLM backbones, particularly in the W3A8 and W2A8 regimes.
Critical weaknesses
-
The layer-wise rank allocation in QLCA (Algorithm 1) is very effective at identifying quantization-friendly compression structures across layers. It would be great if the authors could share a brief breakdown of how many candidate ranks are typically evaluated per layer and the overall offline calibration time for better understanding.
-
In Appendix A.7 (Table 6), the authors provide a helpful latency evaluation showing promising decoding speedups at 10% compression with W4A8. Since decomposing a weight matrix into two low-rank factors replaces one matrix multiplication with two smaller sequential ones, we wonder how the trade-off between parameter reduction and two-layer computation behaves across different compression ratios (e.g., 5% vs. 20%) or on GPU hardware.
-
The results in Tables 1 and 2 show a very interesting trend where combining moderate compression with W3A8 preserves accuracy much better than dropping directly to W2A8. It would be interesting to know how CoCaQ performs if the compression ratio with W3A8 is pushed slightly further (for instance, around 25% to 30% compression) to match the exact memory footprint of W2A8.
Questions for authors
See weakness
3 comments
Response to Reviewer A3vE (Part 2)
Weakness2: In Appendix A.7 (Table 6), the authors provide a helpful latency evaluation showing promising decoding speedups at 10% compression with W4A8. Since decomposing a weight matrix into two low-rank factors replaces one matrix multiplication with two smaller sequential ones, we wonder how the trade-off between parameter reduction and two-layer computation behaves across different compression ratios (e.g., 5% vs. 20%) or on GPU hardware.
Thank you for this insightful comment. We agree that replacing a weight matrix with two low-rank factors introduces an additional matrix multiplication, potentially increasing the number of matrix-loading operations and computation launches. Therefore, the actual latency change depends on the trade-off between the overhead introduced by the additional matrix multiplication and the latency reduction achieved through fewer parameters and reduced computational costs.
To further investigate the trade-off, we conduct experiments to analyze how inference latency varies with the rank of the decomposed matrices. Specifically, we use INT8 to load weight matrices of sizes and , representing attention and MLP layers, respectively. To simulate realistic sequence-length settings, we use an activation matrix of size as input and benchmark inference latency on an NVIDIA RTX A6000 GPU. We evaluate the latency of the original (without low rank decomposition) matrices and progressively reduce the decomposition rank from the upper bound (as specified in our response to Weakness 1) to investigate how latency changes with the rank. The results are presented in Appendix A.16 of the revised manuscript. At higher ranks, low-rank decomposition introduces additional matrix multiplication overhead, resulting in higher latency than the original non-decomposed matrices. As the rank decreases, this overhead gradually diminishes, leading to progressively lower latency that eventually falls below that of the original matrices.
In CoCaQ, QLCA explicitly considers the uncompressed configuration during layer-wise rank selection, allowing it to retain the original structures for selected layers rather than compressing all layers. This effectively limits the overhead introduced by additional matrix multiplications. Combined with the rank-dependent latency trade-off observed in our experiments, this design enables CoCaQ to consistently achieve latency reductions across compression ratios ranging from 5% to 20%, as demonstrated in Appendix A.7 (Table 6).
To further characterize GPU latency, we conduct additional experiments on an NVIDIA RTX A6000 GPU using ExecuTorch[1] to evaluate the decoding latency of CoCaQ. For W3 quantization, we adapt ExecuTorch based on the GPTQ[2] kernel implementation to support W3 inference. Specifically, we measure the average per-token decoding latency when generating 512 tokens after a prefill of 512 input tokens under both W3A8 and W4A8 settings. Each generation run is treated as one measurement, and we repeat the experiment 10 times to obtain the reported results. The results are presented in Table R4. The results show that latency consistently decreases as the compression ratio increases, demonstrating the effectiveness of CoCaQ in reducing decoding latency.
Table R4: Latency of decoding time(ms/token) across different compression ratios.
| Method | Setting | Llama2-7B | Qwen3-8B |
|---|---|---|---|
| SpinQuant | W4A8 | 23.31 | 27.04 |
| CoCaQ | 5%+W4A8 | 23.08 | 26.60 |
| 10%+W4A8 | 22.24 | 25.95 | |
| SpinQuant | W3A8 | 22.74 | 26.48 |
| CoCaQ | 5%+W3A8 | 22.70 | 26.31 |
| 10%+W3A8 | 22.24 | 26.00 | |
| 15%+W3A8 | 21.86 | 25.17 | |
| 20%+W3A8 | 21.37 | 24.90 |
Response to Reviewer A3vE (Part 3)
Weakness3: The results in Tables 1 and 2 show a very interesting trend where combining moderate compression with W3A8 preserves accuracy much better than dropping directly to W2A8. It would be interesting to know how CoCaQ performs if the compression ratio with W3A8 is pushed slightly further (for instance, around 25% to 30% compression) to match the exact memory footprint of W2A8.
We thank the reviewer for this comment. To further investigate the potential of combining more aggressive compression with W3A8 quantization, we extend the compression ratios to 25% and 30% and report the results in Table R5, matching the memory footprint of W2A8 quantization.
Overall, CoCaQ consistently outperforms FlatQuant[8] and SERQ[7], while its performance relative to SpinQuant[6] varies across models. For Llama2-7B, CoCaQ combining W3A8 quantization with compression consistently outperforms SpinQuant under W2A8 quantization at both compression ratios. For Qwen3-8B, CoCaQ combining more aggressive compression with W3A8 quantization exhibits a slight performance drop compared with SpinQuant under W2A8 quantization. This may be partly attributed to the varying sensitivity of different model architectures to compression. As shown in Table R6, certain architectures, such as Qwen3-8B, are sensitive to compression and suffer considerable performance degradation even before quantization. This further motivates our approach of incorporating quantization at relatively high compression ratios to improve the overall robustness of the compressed model. Nevertheless, exploring more effective compression methods could further improve CoCaQ's performance in future work.
Table R5: Performance of CoCaQ with extended compression under W3A8.
| Memory(GB)↓ | Memory(GB)↓ | CS Avg.↑ | CS Avg.↑ | |||
|---|---|---|---|---|---|---|
| Type | Method | Setting | L2-7B | Q3-8B | L2-7B | Q3-8B |
| C | SVD-LLM | 85% | 2.46 | 4.57 | 34.84 | 33.94 |
| KFAC-SVD* | 85% | 2.46 | 4.57 | 34.73 | 34.90 | |
| Q | SpinQuant | W2A8 | 2.24 | 4.33 | 43.13 | 52.78 |
| FlatQuant | W2A8 | 2.24 | 4.33 | 35.18 | 34.36 | |
| SERQ | W2A8 | 2.28 | 4.37 | 36.62 | 35.47 | |
| C+Q | CoCaQ (Ours) | 20%+W3A8 | 2.55 | 4.66 | 53.02 | 54.30 |
| 25%+W3A8 | 2.42 | 4.52 | 48.39 | 47.24 | ||
| 30%+W3A8 | 2.27 | 4.39 | 44.71 | 39.44 |
Table R6: Comparison of CoCaQ and competing methods under compression-only and joint compression–quantization settings on Qwen3-8B.
| Method | Setting | Wiki2↓ | CS Avg.↑ |
|---|---|---|---|
| Baseline | 7.00 | 68.62 | |
| SVD-LLM | 25% | 11.93 | 55.29 |
| 25% + W3A8 | 25.26 | 40.21 | |
| 30% | 14.66 | 47.83 | |
| 30% + W3A8 | 39.58 | 36.45 | |
| UniQL | 25% | 18.62 | 45.10 |
| 25% + W3A8 | 31.41 | 40.65 | |
| 30% | 22.96 | 43.22 | |
| 30% + W3A8 | 41.25 | 38.71 | |
| CoCaQ (Ours) | 25% | 11.76 | 54.87 |
| 25% + W3A8 | 18.55 | 47.24 | |
| 30% | 15.09 | 50.14 | |
| 30% + W3A8 | 27.70 | 39.44 |
Response to Reviewer A3vE (Part 1)
We thank the reviewer for the positive feedback. Detailed responses to the reviewer’s questions and concerns are provided below. We look forward to constructive discussions to further improve our work.
We sincerely thank the reviewer for this helpful suggestion. We are happy to provide further details about the candidate rank selection and offline calibration cost of QLCA.
For an original weight matrix W∈Rm×n, its low-rank decomposition produces two factor matrices of sizes m×k and k×n, respectively. To achieve an actual parameter reduction, the rank k must satisfy k(m+n)<mn, thus we have k<m+nmn. To improve search efficiency, we sample candidate ranks at intervals of 256: K={256j∣256j<k∗,j∈N+}, where k∗=m+nmn. Consequently, the number of candidate ranks is determined adaptively for each layer according to its matrix dimensions, rather than using a fixed search space for all layers.
We evaluate the overall offline calibration time for Llama2-7B on 8 NVIDIA RTX A6000 GPUs. The complete offline calibration of QLCA takes approximately 1 hour, including layer-wise candidate evaluation and rank selection. Moreover, the layer-wise candidate sensitivity scores can be stored and reused for different compression ratios, avoiding redundant candidate evaluations. Consequently, determining the compressed structure for a new compression ratio incurs only 0.03 seconds of additional search time, averaged over 100 independent runs to reduce measurement variability. We hope these details provide a clearer understanding of the practical computational cost of QLCA.