acceptodds
Under review as a conference paper at ICLR 2027

CoCaQ: Coupling of Compression and Quantization for Flexible LLM Deployment

Abstract

The practical deployment of Large Language Models (LLMs), especially on edge devices, is heavily constrained by their massive model sizes and prohibitive computation costs, which are generally addressed by either model compression or quantization. In practice, these two kinds of techniques can be applied concurrently for flexible LLM deployment, especially under extremely low memory budgets, which, however, are typically optimized separately and so lead to suboptimal performance. Particularly, we observe that the joint of model compression and quantization is challenged by the effect of compressed structures on quantization and the ability of quantization to account for distribution shifts induced by compression. To address the above issues, we propose a new Coupling of Compression and Quantization () method to perform model compression and quantization in a coupling manner, aiming to achieve effective and flexible LLM deployment by unifying the two techniques. To this end, our CoCaQ is developed with two key components, including a quantization-aware model compression (QaMC) and an activation stabilization for quantization (ASQ). First, our QaMC constructs quantization-friendly low-rank representations based on activation statistics and optimizes the layer-wise compressed structures with a quantization-aware loss, aiming to identify the robust substructures of LLM for the subsequent quantization. Furthermore, our ASQ introduces an efficient orthogonal rotation regularization for low-rank weight matrices, enabling low-bit quantization to better tolerate activation amplification induced by the preceding model compression. Based on the proposed QaMC and ASQ, our CoCaQ couples model compression and quantization by fully considering the mutual influence between the two, resulting in an effective yet flexible solution for LLM deployment. Experiments are conducted across LLaMA and Qwen models, and the results show that our CoCaQ achieves better memory-accuracy trade-offs than state-of-the-art compression-only and quantization-only methods, while clearly outperforming the counterparts with joint of compression and quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, model quantization, model compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A3vEOpenReview ↗
Rating
4
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCaQ, a training-free framework that couples low-rank SVD weight compression and low-bit post-training quantization (PTQ) for flexible LLM deployment under tight memory budgets. The authors observe two key issues when combining compression and quantization sequentially: 1) layer-wise rank structures that are optimal before quantization do not maintain their relative ranking after low-bit quantization, and 2) decomposing a weight matrix W into low-rank factors A_r and B_r introduces an intermediate activation H = B_r X with severe outliers that degrade low-bit activation quantization. To solve these issues, CoCaQ introduces Quantization-aware Model Compression (QaMC)—combining Quantization-Friendly SVD (QF-SVD) with Quantization-aware Layer-wise Compression Allocation (QLCA) based on post-quantization output KL divergence—and Activation Stabilization for Quantization (ASQ), which inserts an orthogonal rotation matrix R* between the low-rank factors to smooth intermediate activation outliers. Experiments on LLaMA-2, LLaMA-3, and Qwen3 models demonstrate improved accuracy-memory trade-offs over standalone compression, standalone quantization, and sequential or joint compression-plus-quantization baselines.

Critical strengths
  1. The paper clearly identifies and empirically illustrates (Figure 2) two concrete bottlenecks in combining low-rank compression with quantization: the rank-order shift of layer-wise structures after quantization and the emergence of activation outliers in the intermediate low-rank bottleneck H = B_r X.

  2. The proposed training-free solutions—Frobenius norm balancing in QF-SVD (Equation 8), orthogonal rotation smoothing across the low-rank factors in ASQ (Equation 12), and post-quantization KL rank search in QLCA—are mathematically clean and achieve strong gains over SVD-LLM, KFAC-SVD, and UniQL across five LLM backbones, particularly in the W3A8 and W2A8 regimes.

Critical weaknesses
  1. The layer-wise rank allocation in QLCA (Algorithm 1) is very effective at identifying quantization-friendly compression structures across layers. It would be great if the authors could share a brief breakdown of how many candidate ranks are typically evaluated per layer and the overall offline calibration time for better understanding.

  2. In Appendix A.7 (Table 6), the authors provide a helpful latency evaluation showing promising decoding speedups at 10% compression with W4A8. Since decomposing a weight matrix into two low-rank factors replaces one matrix multiplication with two smaller sequential ones, we wonder how the trade-off between parameter reduction and two-layer computation behaves across different compression ratios (e.g., 5% vs. 20%) or on GPU hardware.

  3. The results in Tables 1 and 2 show a very interesting trend where combining moderate compression with W3A8 preserves accuracy much better than dropping directly to W2A8. It would be interesting to know how CoCaQ performs if the compression ratio with W3A8 is pushed slightly further (for instance, around 25% to 30% compression) to match the exact memory footprint of W2A8.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer A3vE (Part 1)

We thank the reviewer for the positive feedback. Detailed responses to the reviewer’s questions and concerns are provided below. We look forward to constructive discussions to further improve our work.

Weakness1: The layer-wise rank allocation in QLCA (Algorithm 1) is very effective at identifying quantization-friendly compression structures across layers. It would be great if the authors could share a brief breakdown of how many candidate ranks are typically evaluated per layer and the overall offline calibration time for better understanding.

We sincerely thank the reviewer for this helpful suggestion. We are happy to provide further details about the candidate rank selection and offline calibration cost of QLCA.

For an original weight matrix W∈Rm×n\mathbf{W}\in\mathbb{R}^{m\times n}, its low-rank decomposition produces two factor matrices of sizes m×km\times k and k×nk\times n, respectively. To achieve an actual parameter reduction, the rank kk must satisfy k(m+n)<mnk(m+n)<mn, thus we have k<mnm+nk< \frac{mn}{m+n}. To improve search efficiency, we sample candidate ranks at intervals of 256: K={256j∣256j<k∗,j∈N+}K = \lbrace 256j | 256j < k^{\ast}, j \in \mathbb{N}^{+} \rbrace, where k∗=mnm+nk^{\ast}=\frac{mn}{m+n}. Consequently, the number of candidate ranks is determined adaptively for each layer according to its matrix dimensions, rather than using a fixed search space for all layers.

We evaluate the overall offline calibration time for Llama2-7B on 8 NVIDIA RTX A6000 GPUs. The complete offline calibration of QLCA takes approximately 1 hour, including layer-wise candidate evaluation and rank selection. Moreover, the layer-wise candidate sensitivity scores can be stored and reused for different compression ratios, avoiding redundant candidate evaluations. Consequently, determining the compressed structure for a new compression ratio incurs only 0.03 seconds of additional search time, averaged over 100 independent runs to reduce measurement variability. We hope these details provide a clearer understanding of the practical computational cost of QLCA.

AuthorsOpenReview ↗

Response to Reviewer A3vE (Part 2)

Weakness2: In Appendix A.7 (Table 6), the authors provide a helpful latency evaluation showing promising decoding speedups at 10% compression with W4A8. Since decomposing a weight matrix into two low-rank factors replaces one matrix multiplication with two smaller sequential ones, we wonder how the trade-off between parameter reduction and two-layer computation behaves across different compression ratios (e.g., 5% vs. 20%) or on GPU hardware.

Thank you for this insightful comment. We agree that replacing a weight matrix with two low-rank factors introduces an additional matrix multiplication, potentially increasing the number of matrix-loading operations and computation launches. Therefore, the actual latency change depends on the trade-off between the overhead introduced by the additional matrix multiplication and the latency reduction achieved through fewer parameters and reduced computational costs.

To further investigate the trade-off, we conduct experiments to analyze how inference latency varies with the rank of the decomposed matrices. Specifically, we use INT8 to load weight matrices of sizes 4096×40964096 \mathbin{\times} 4096 and 4096×110084096\mathbin{\times}11008, representing attention and MLP layers, respectively. To simulate realistic sequence-length settings, we use an activation matrix of size 2048×40962048\mathbin{\times}4096 as input and benchmark inference latency on an NVIDIA RTX A6000 GPU. We evaluate the latency of the original (without low rank decomposition) matrices and progressively reduce the decomposition rank from the upper bound k∗k^{\ast} (as specified in our response to Weakness 1) to investigate how latency changes with the rank. The results are presented in Appendix A.16 of the revised manuscript. At higher ranks, low-rank decomposition introduces additional matrix multiplication overhead, resulting in higher latency than the original non-decomposed matrices. As the rank decreases, this overhead gradually diminishes, leading to progressively lower latency that eventually falls below that of the original matrices.

In CoCaQ, QLCA explicitly considers the uncompressed configuration during layer-wise rank selection, allowing it to retain the original structures for selected layers rather than compressing all layers. This effectively limits the overhead introduced by additional matrix multiplications. Combined with the rank-dependent latency trade-off observed in our experiments, this design enables CoCaQ to consistently achieve latency reductions across compression ratios ranging from 5% to 20%, as demonstrated in Appendix A.7 (Table 6).

To further characterize GPU latency, we conduct additional experiments on an NVIDIA RTX A6000 GPU using ExecuTorch[1] to evaluate the decoding latency of CoCaQ. For W3 quantization, we adapt ExecuTorch based on the GPTQ[2] kernel implementation to support W3 inference. Specifically, we measure the average per-token decoding latency when generating 512 tokens after a prefill of 512 input tokens under both W3A8 and W4A8 settings. Each generation run is treated as one measurement, and we repeat the experiment 10 times to obtain the reported results. The results are presented in Table R4. The results show that latency consistently decreases as the compression ratio increases, demonstrating the effectiveness of CoCaQ in reducing decoding latency.

Table R4: Latency of decoding time(ms/token) across different compression ratios.

MethodSettingLlama2-7BQwen3-8B
SpinQuantW4A823.3127.04
CoCaQ5%+W4A823.0826.60
10%+W4A822.2425.95
SpinQuantW3A822.7426.48
CoCaQ5%+W3A822.7026.31
10%+W3A822.2426.00
15%+W3A821.8625.17
20%+W3A821.3724.90
AuthorsOpenReview ↗

Response to Reviewer A3vE (Part 3)

Weakness3: The results in Tables 1 and 2 show a very interesting trend where combining moderate compression with W3A8 preserves accuracy much better than dropping directly to W2A8. It would be interesting to know how CoCaQ performs if the compression ratio with W3A8 is pushed slightly further (for instance, around 25% to 30% compression) to match the exact memory footprint of W2A8.

We thank the reviewer for this comment. To further investigate the potential of combining more aggressive compression with W3A8 quantization, we extend the compression ratios to 25% and 30% and report the results in Table R5, matching the memory footprint of W2A8 quantization.

Overall, CoCaQ consistently outperforms FlatQuant[8] and SERQ[7], while its performance relative to SpinQuant[6] varies across models. For Llama2-7B, CoCaQ combining W3A8 quantization with compression consistently outperforms SpinQuant under W2A8 quantization at both compression ratios. For Qwen3-8B, CoCaQ combining more aggressive compression with W3A8 quantization exhibits a slight performance drop compared with SpinQuant under W2A8 quantization. This may be partly attributed to the varying sensitivity of different model architectures to compression. As shown in Table R6, certain architectures, such as Qwen3-8B, are sensitive to compression and suffer considerable performance degradation even before quantization. This further motivates our approach of incorporating quantization at relatively high compression ratios to improve the overall robustness of the compressed model. Nevertheless, exploring more effective compression methods could further improve CoCaQ's performance in future work.

Table R5: Performance of CoCaQ with extended compression under W3A8.

Memory(GB)↓Memory(GB)↓CS Avg.↑CS Avg.↑
TypeMethodSettingL2-7BQ3-8BL2-7BQ3-8B
CSVD-LLM85%2.464.5734.8433.94
KFAC-SVD*85%2.464.5734.7334.90
QSpinQuantW2A82.244.3343.1352.78
FlatQuantW2A82.244.3335.1834.36
SERQW2A82.284.3736.6235.47
C+QCoCaQ (Ours)20%+W3A82.554.6653.0254.30
25%+W3A82.424.5248.3947.24
30%+W3A82.274.3944.7139.44

Table R6: Comparison of CoCaQ and competing methods under compression-only and joint compression–quantization settings on Qwen3-8B.

MethodSettingWiki2↓CS Avg.↑
Baseline7.0068.62
SVD-LLM25%11.9355.29
25% + W3A825.2640.21
30%14.6647.83
30% + W3A839.5836.45
UniQL25%18.6245.10
25% + W3A831.4140.65
30%22.9643.22
30% + W3A841.2538.71
CoCaQ (Ours)25%11.7654.87
25% + W3A818.5547.24
30%15.0950.14
30% + W3A827.7039.44
Reviewer_LuiBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCaQ, a training-free framework that couples low-rank compression and quantization for memory-efficient LLM deployment. The method introduces quantization-friendly low-rank decomposition, quantization-aware layer-wise rank allocation, and activation stabilization through orthogonal rotations. Experiments across multiple LLMs demonstrate favorable memory–accuracy trade-offs compared with existing approaches.

Critical strengths
  1. Clear motivation and coherent method design. The paper identifies meaningful challenges in combining compression and quantization. The insights are closely connected to the proposed method, with each component directly addressing an identified issue.
  2. Comprehensive and convincing experiments. The evaluation covers multiple model families, compression ratios, and quantization settings. The comparisons with relevant baselines and ablation studies provide solid evidence for the effectiveness of the proposed framework.
Critical weaknesses
  1. Limited methodological novelty. The observations are relatively intuitive, and the proposed components largely build upon established techniques in model compression and quantization. The overall contribution appears more incremental than fundamentally novel.
  2. Unclear compression–quantization configuration selection. Although the framework emphasizes flexible deployment under different memory budgets, the procedure for selecting the compression ratio and quantization bit-width is not sufficiently explained.
Questions for authors
  1. How are the compression ratio and quantization bit-width selected for a given memory budget? Does the framework determine them automatically?
  2. Could the authors further clarify the main methodological novelty of CoCaQ beyond existing compression and quantization techniques?
4 comments
AuthorsOpenReview ↗

Response to Reviewer LuiB (Part 1)

We thank the reviewer for their time and effort in evaluating our paper. Below, we provide detailed responses to each question and weakness. We look forward to a constructive discussion with the reviewer and to further clarify and justify the assigned ratings.

Weakness1: Limited methodological novelty. The observations are relatively intuitive, and the proposed components largely build upon established techniques in model compression and quantization. The overall contribution appears more incremental than fundamentally novel.

Question2: Could the authors further clarify the main methodological novelty of CoCaQ beyond existing compression and quantization techniques?

We thank the reviewer for raising this important concern. The core contribution of CoCaQ lies in addressing the challenges of joint model compression and quantization for flexible LLM deployment, including the impact of compressed structures on quantization and the requirement of quantization to accommodate compression-induced distribution shifts. Existing methods typically optimize compression and quantization independently or apply them sequentially[2, 3, 4, 6, 7, 8]. UniQL[5] further achieves quantization-friendly SVD, but applies this approach only to v_proj and o_proj in attention module, which has limited contribution to final memory decrease and so the performance. Clearly distinguished from existing works, our CoCaQ makes the first attempt to develop a unified framework that explicitly couples model compression and quantization, motivated by our analysis on their mutual influence. From the methodological perspective, CoCaQ presents a quantization-aware model compression (QaMC) to address the structure sensitivity of post-quantization performance, while activation stabilization for quantization (ASQ) is introduced to mitigate compression-induced shifts in activation distributions.

Specifically, (1) QaMC considers both compression and quantization losses to derive quantization-aware compressed models. Within QaMC, QF-SVD exploits output activation statistics to derive quantization-friendly low-rank representations across all layers, with its effectiveness theoretically and empirically validated through the analysis of reconstruction and quantization errors in Appendix A.1 and A.2. Besides, different from conventional compression strategies that determine model structures based on pre-quantization metrics[3], QLCA selects quantization-friendly compressed structures based on post-quantization sensitivity, motivated by the observation in Fig2.(a). To the best of our knowledge, QLCA makes the first attempt to explicitly incorporate quantization error into the search for compressed model structures. Its effectiveness is further validated by substantial performance gains in the ablation studies (Table 3). (2) Different from existing methods that primarily address outliers in the original model[6], ASQ makes the first attempt to explicitly address the additional outliers and activation distribution shifts introduced by low-rank decomposition, explicitly mitigating the adverse impact of compression on subsequent quantization and improving post-quantization robustness.

AuthorsOpenReview ↗

Response to Reviewer LuiB (Part 2)

To further clarify the difference from existing methods, we present a comparison in Table R3 to highlight the methodological novelty of CoCaQ over existing methods that combine model compression and quantization.

Table R3: Comparison of CoCaQ with existing methods that combine model compression and quantization.

MethodCombination StrategyDegree of CouplingCompression ParadigmQuantization-aware Model CompressionCompression-Aware Quantization
KFAC-SVD*SequentialLowweight-based
SVD-LLMSequentialLowinput-based
UniQLSequential (Integrated into SVD)Medium(input-aware) weight-based✔️/✘
CoCaQ(Ours)Joint designHighoutput-based✔️✔️

Besides the above methodological contribution, our CoCaQ involves contributions from the perspectives of task, insight, and practical application. From the task perspective, CoCaQ focuses on coupling model compression and quantization, achieving flexible memory allocation while preserving model performance well, enabling more flexible and practical LLM deployment than those achievable by either standalone paradigm. From the insight perspective, CoCaQ is, to the best of our knowledge, the first to investigate the mutual influence between model compression and quantization, revealing the sensitivity of quantization to compressed structures and the additional activation outliers and distribution shifts induced by low-rank decomposition. It potentially provides valuable insights for future research on coupling model compression and quantization. From the perspective of practical application, CoCaQ enables more flexible memory allocation and more favorable accuracy–memory trade-offs, facilitating the deployment of LLMs on resource-constrained devices. Overall, CoCaQ provides a new method for joint model compression and quantization, providing a promising direction for future research on flexible LLM deployment.

AuthorsOpenReview ↗

Response to Reviewer LuiB (Part 3)

Weakness2: Unclear compression–quantization configuration selection. Although the framework emphasizes flexible deployment under different memory budgets, the procedure for selecting the compression ratio and quantization bit-width is not sufficiently explained.

Question1: How are the compression ratio and quantization bit-width selected for a given memory budget? Does the framework determine them automatically?

We thank the reviewer for raising this important question. We apologize for not clarifying this aspect in the original manuscript. For a fair comparison, all methods are evaluated under identical combinations of compression ratios and quantization bit-widths, with their corresponding memory footprints reported accordingly. To address this concern, we adopt a simple automatic configuration selection strategy. Given a target memory budget, we first estimate the equivalent average bit-width based on the original model size. We then consider the two neighboring integer bit-widths. For example, for an equivalent budget of 3.5 bits, we evaluate 3-bit quantization and 4-bit quantization combined with an appropriate compression ratio to satisfy the same memory budget. We finally compare their final model performance and select the better configuration.

This strategy provides a straightforward way to automatically determine the compression ratio and quantization bit-width under a given memory budget, without requiring additional training. We further clarify this configuration selection procedure and provide the corresponding pseudocode in Appendix A.15 of the revised manuscript.

Algorithm: Automatic Configuration Selection under a Memory Budget
Input: Model MM with PP parameters, target memory budget BB
Output: Selected configuration C∗=(q∗,r∗)C^{\ast}=(q^{\ast},r^{\ast}), where q∗q^{\ast} and r∗r^{\ast} denote the quantization bit-width and compression ratio, respectively
Compute the equivalent bit-width beq=8∗B/Pb_{{eq}} = {8*B}/{P}
The lower bit-width qlow=⌊beq⌋q_{{low}} = \lfloor b_{{eq}} \rfloor
The upper bit-width qhigh=⌈beq⌉q_{{high}} = \lceil b_{{eq}} \rceil
Construct candidate configuration C1=(qlow,0)C_1 = (q_{{low}}, 0)
For qhighq_{{high}} seek compression rate rhigh=1−beq/qhighr_{{high}} =1-{b_{{eq}}}/{q_{{high}}}
Construct candidate configuration C2=(qhigh,rhigh)C_2 = (q_{{high}}, r_{{high}})
Apply C1C_1 and C2C_2 to MM and evaluate the performance of both configurations
Select the configuration with better performance C∗=arg⁡max⁡C∈{C1,C2}Performance⁡(M,C)C^{\ast}=\arg\max_{{C}\in\{C_1, C_2\}}\operatorname{Performance}(M, C)
Return C∗C^{\ast}
AuthorsOpenReview ↗

Discussion period reminder

Dear Reviewer LuiB,

Thank you again for your careful review and constructive feedback. We have carefully considered your comments, addressed them in our rebuttal, and revised the manuscript accordingly.

As the discussion period draws to a close, we would greatly appreciate your further feedback on whether our responses and revisions have adequately resolved your main concerns. If any questions remain, we would be happy to provide additional clarification and discussion immediately.

Thank you again for your valuable time, effort, and support!

Best regards,

The Authors

Reviewer_9UGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CoCaQ, which effectively combines quantization and low-rank approximation to compress LLMs. They achieves the best memory-accuracy trade-off among quantization, low-rank approximation, and hybrid methods when they compress LLaMA models.

Critical strengths
  • When determining the compression ratio of each layer, the method measures sensitivity after actually applying quantization and reflects it in the allocation, which leads to high accuracy after compression.
  • CoCaQ outperforms existing hybrid methods, and the performance gap becomes larger as the compression ratio increases and when compressing Qwen3-8B.
Critical weaknesses
  • While the paper analyzes the memory usage of the compressed models, it does not provide any analysis of inference speed on GPUs.
  • Layer-wise compression ratio after measuring sensitivity after quantization, which is the main part of the CoCaQ, is a not algorithmically novel approach.
Questions for authors
  • Could you further clarify the novelty of CoCaQ compared to existing methods?
  • Could you provide latency analysis results for CoCaQ and compare the latency-accuracy curve with competitors?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 9UGd (Part 1)

We thank the reviewer for their time and effort in evaluating our paper. Below, we provide detailed responses to each question and weakness. We look forward to a constructive discussion with the reviewer and to further clarify and justify the assigned ratings.

Weakness1: While the paper analyzes the memory usage of the compressed models, it does not provide any analysis of inference speed on GPUs.

Question2: Could you provide latency analysis results for CoCaQ and compare the latency-accuracy curve with competitors?

Thank you for this comment. We kindly note that the original manuscript reported CPU inference latency in Appendix A.7 (Table 6), showing that compression beyond quantization further reduces the inference latency. To provide a more comprehensive comparison, we conduct additional experiments on an NVIDIA RTX A6000 GPU using ExecuTorch[1] to compare the decoding latency of CoCaQ with competing methods. For W3 quantization, we adapt ExecuTorch based on the GPTQ[2] kernel implementation to support W3 inference. Specifically, we measure the average per-token decoding latency when generating 512 tokens after a prefill of 512 input tokens under both W3A8 and W4A8 settings. Each generation run is treated as one measurement, and we repeat the experiment 10 times to obtain the reported results. The results are presented in Table R1, and the corresponding latency–accuracy curves for GPU are provided in Appendix A.14 of the revised manuscript.

The latency–accuracy curves demonstrate that CoCaQ achieves a favorable latency–accuracy trade-off across both Llama2-7B and Qwen3-8B. Compared with KFAC-SVD*[3] and SVD-LLM[4], CoCaQ consistently delivers higher accuracy at comparable decoding latency. Moreover, under the same memory budget, CoCaQ achieves lower latency than them. Compared with UniQL[5], CoCaQ consistently achieves higher accuracy at comparable latency on Qwen3-8B. On LLaMA2-7B, CoCaQ also outperforms UniQL at higher latency levels, while at lower latency levels, it achieves slightly lower accuracy but offers additional memory reduction. Overall, these results demonstrate that CoCaQ provides a competitive latency–accuracy trade-off while effectively balancing memory consumption and model accuracy.

In terms of latency, across different compression ratios, CoCaQ consistently outperforms SVD-LLM, achieves comparable latency to KFAC-SVD* , and exhibits only a slight latency increase of a few milliseconds over UniQL, while consistently achieving the best accuracy. We analyze that the latency difference is primarily associated with the compressed model structures adopted by these methods. In comparison, SVD-LLM applies low-rank decomposition to all layers, introducing additional matrix multiplications more broadly, as shown in the visualization of the compressed structure in Appendix A.14. KFAC-SVD* adopts a compressed structure similar to CoCaQ, resulting in comparable inference latency. As for UniQL, it combines channel pruning with low-rank decomposition, and the decomposition is applied only to two types of layers in the attention module. Therefore, low-rank decomposition accounts for a relatively small proportion of the overall compression, while most of the memory reduction comes from channel pruning. Unlike low-rank decomposition, channel pruning directly reduces the number of channels without introducing additional matrix multiplications, thereby reducing inference latency as the compression ratio increases. However, this comes at the cost of reduced generalization capability, as reflected in the results in Table 2 and the performance trends in Appendix A.14. Overall, CoCaQ achieves a favorable trade-off between inference latency and model performance, while offering flexibility in balancing memory usage and performance.

AuthorsOpenReview ↗

Response to Reviewer 9UGd (Part 2)

Table R1: Decoding latency(ms/token) comparison across different methods on GPU.

Llama2-7BQwen3-8B
SettingMethodDecoding time(ms/token) ↓CS Avg. ↑Decoding time(ms/token) ↓CS Avg. ↑
5%+W4A8KFAC-SVD*23.3962.3427.1463.81
SVD-LLM25.1249.0929.8458.71
UniQL22.0962.6326.2868.35
CoCaQ (Ours)23.0863.3026.6068.77
10%+W4A8KFAC-SVD*23.1962.4626.7959.26
SVD-LLM24.0947.0129.0056.13
UniQL21.2260.9824.2855.01
CoCaQ (Ours)22.2463.4925.9567.57
5%+W3A8KFAC-SVD*23.1757.6726.7951.26
SVD-LLM25.0340.5428.9848.46
UniQL22.7357.5725.7060.62
CoCaQ (Ours)22.7060.2326.3165.76
10%+W3A8KFAC-SVD*22.8155.3526.5549.26
SVD-LLM24.0938.3428.5648.24
UniQL21.0056.2824.3545.90
CoCaQ (Ours)22.2459.8726.0062.79
15%+W3A8KFAC-SVD*22.6252.5326.3047.90
SVD-LLM23.6438.3827.2943.28
UniQL19.8153.6523.3545.74
CoCaQ (Ours)21.8657.0525.1758.07
20%+W3A8KFAC-SVD*22.0549.8425.5445.68
SVD-LLM23.1737.3026.6841.59
UniQL18.8050.8622.0442.84
CoCaQ (Ours)21.3753.0224.9054.30
AuthorsOpenReview ↗

Response to Reviewer 9UGd (Part 3)

Weakness2: Layer-wise compression ratio after measuring sensitivity after quantization, which is the main part of the CoCaQ, is a not algorithmically novel approach.

Question1: Could you further clarify the novelty of CoCaQ compared to existing methods?

While we remain open to constructive discussion with the reviewer, we highlight our novelty below and further clarify why CoCaQ goes beyond existing methods. The novelty of CoCaQ lies in four aspects. (1) Task: CoCaQ couples model compression and quantization, achieving flexible memory allocation while preserving model performance well. (2) Insight: We analyze the mutual influence between model compression and quantization. (3) Technique: Motivated by the insight, we develop complementary modules of quantization-friendly model compression (QaMC) and activation stabilization for quantization (ASQ). (4) Practice: CoCaQ establishes a practical approach to couple model compression and quantization, enabling more effective accuracy–memory trade-offs and facilitating the real-world application of LLMs on resource-constrained devices.

(1) Task: Existing methods for efficient LLM deployment can be broadly classified into two categories, including model compression and quantization[3, 4, 6, 7, 8]. Model compression[3, 4] reduces the number of parameters, but aggressive compression often leads to substantial performance degradation, as shown in Fig. 1 and Table 1. In contrast, model quantization[6, 7, 8] reduces memory consumption by lowering the bit-width while largely preserving model performance. However, uniform bit-widths provide discrete memory configurations, making it difficult to accommodate flexible memory budgets(e.g., the whitespace in memory of 2.5GB~3GB of SpinQuant[6] and SERQ[7] in Fig.1(left)). Although mixed-precision quantization can provide finer-grained memory allocation, it may introduce irregular computation patterns and require specialized hardware support or customized kernels to maintain inference efficiency[9].

To address these limitations, CoCaQ couples model compression and quantization, achieving flexible memory allocation while preserving model performance well. This enables better memory-performance trade-offs beyond those achievable by either standalone paradigm, as demonstrated in Fig. 1. To further clarify the difference to existing methods, we present a comparison in Table R2 to highlight the novelty of CoCaQ over existing single-paradigm methods.

Table R2: Comparison of CoCaQ with existing single-paradigm methods.

MethodQuantizationCompressionMemory FlexibilityPerformance under Low Memory
SVD-LLM✔️✔️
KFAC-SVD*✔️✔️
SpinQuant✔️✔️
FlatQuant✔️✔️
SERQ✔️✔️
CoCaQ(Ours)✔️✔️✔️✔️

(2) Insight: To the best of our knowledge, CoCaQ is the first to analyze the mutual influence between model compression and quantization. We find that post-quantization performance is sensitive to the compressed structures (Fig. 2(a)), while low-rank decomposition introduces additional outliers that shift the activation distribution (Fig. 2(b)), providing valuable insights for future research on coupling model compression and quantization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their time and effort in evaluating our manuscript. We appreciate the thoughtful feedback, which has helped us further strengthen the work, and we have revised the manuscript accordingly. The newly added content in the revised manuscript is marked in blue and listed below in the L1.

In response to the reviewers’ feedback, we have added all requested experiments and comparisons, which substantially strengthen the work. All additional experimental tables introduced in the discussion are labeled with the prefix “R” (e.g., Table R1, Table R2) as listed in the L2, while tables without the “R” prefix refer to those already included in the manuscript.

L1: List of modification to our manuscript

  • Add Appendix [A.14 Latency Comparison with Competitors] (reviewer 9UGd)

  • Add Appendix [A.15 Automatic Configuration Selection under a Memory Budget] (reviewer LuiB)

  • Add Appendix [A.16 Trade-off Between Decomposition Rank and Inference Latency] (reviewer A3vE)

L2: List of tables used in the discussion

  • Table R1: Decoding latency(ms/token) comparison across different methods on GPU.
  • Table R2: Comparison of CoCaQ with existing single-paradigm methods.
  • Table R3: Comparison of CoCaQ with existing methods that combine model compression and quantization.
  • Table R4: Latency of decoding time(ms/token) across different compression ratios.
  • Table R5: Performance of CoCaQ with extended compression under W3A8.
  • Table R6: Comparison of CoCaQ and competing methods under compression-only and joint compression–quantization settings on Qwen3-8B.
  • Table R7: Robustness of QLCA-selected compressed structures to low-bit quantization.
  • Table R8: Comparison across different methods at compression ratios below 5% on Qwen3-8B.

L3: List of references used in the discussion

[1] Nachin, Mergen, et al. "ExecuTorch-A Unified PyTorch Solution to Run ML Models On-Device." Proceedings of Machine Learning and Systems 8 (2026): 569-609.

[2] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." International Conference on Learning Representations. Vol. 2023. 2023.

[3] Thoma, Moritz, et al. "Advancing SVD-based LLM Compression via Layer-Wise Error Model Search." Forty-third International Conference on Machine Learning. 2026.

[4] Wang, Xin, et al. "Svd-llm: Truncation-aware singular value decomposition for large language model compression." International Conference on Learning Representations. Vol. 2025. 2025.

[5] Chiang, Hung-Yueh, et al. "UniQL: Unified quantization and low-rank compression for adaptive edge LLMs." International Conference on Learning Representations. Vol. 2026. 2026.

[6] Liu, Zechun, et al. "Spinquant: Llm quantization with learned rotations." International Conference on Learning Representations. Vol. 2025. 2025.

[7] Park, Yeonsik, Hyeonseong Kim, and Seungkyu Choi. "SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization." International Conference on Learning Representations. Vol. 2026. 2026.

[8] Sun, Yuxuan, et al. "Flatquant: Flatness matters for llm quantization." Forty-second International Conference on Machine Learning. 2025.

[9] Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving." Proceedings of Machine Learning and Systems 6 (2024): 196-209.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.