- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors explore a novel approach that combines weights quantized by two different quantization methods, AWQ and GPTQ, to obtain a single quantized model, and they report performance improvements from this approach.
Critical strengths
To the best of my knowledge, the idea of combining the results of two different quantization methods has not been explored before. The main strength of this paper is to propose such novel idea and verfying it with experiments.
Critical weaknesses
The idea of the paper is interesting, but the following aspects need to be improved.
- Lack of experiments with recent models, methods, and benchmarks.
- Marginal performance gain
- Absence of latency analysis.
- The authors use element-wise weight reconstruction errors for defining the fraction of weights for which each method is more accurate in Figure 1, while GPTQ's objective function includes the distribution of activations.
Questions for authors
Could you provide your responses to the points raised in the Weakness section?
4 comments
Response to Q3: Native quantized inference latency and GPU memory
Thank you for highlighting the need for latency analysis. We have implemented packed 3-bit and 4-bit lookup-table kernels for EGComp-AG and added native quantized inference measurements across four models at both bit widths.
Table R3. Native quantized inference latency and peak GPU memory for the quantization configurations in Table R1.
| Model | Bits | Method | TTFT (ms) ↓ | Decode (ms/token) ↓ | Peak GPU memory (GiB) ↓ |
|---|---|---|---|---|---|
| LLaMA-1-7B | 3 | SqueezeLLM | 19631.942 | 30.780 | 4.110 |
| 3 | EGComp-AG | 258.911 | 11.165 | 4.028 | |
| 4 | SqueezeLLM | 19682.456 | 31.662 | 4.973 | |
| 4 | EGComp-AG | 286.869 | 10.355 | 4.874 | |
| OPT-6.7B | 3 | SqueezeLLM | 15774.171 | 21.163 | 4.820 |
| 3 | EGComp-AG | 246.999 | 9.801 | 4.802 | |
| 4 | SqueezeLLM | 16080.239 | 21.783 | 5.634 | |
| 4 | EGComp-AG | 269.567 | 9.179 | 5.599 | |
| Qwen3-8B-Base | 3 | SqueezeLLM | 14293.978 | 24.766 | 5.473 |
| 3 | EGComp-AG | 294.070 | 14.283 | 5.382 | |
| 4 | SqueezeLLM | 15065.183 | 26.621 | 6.342 | |
| 4 | EGComp-AG | 316.913 | 13.943 | 6.228 | |
| Mistral-7B-v0.1 | 3 | SqueezeLLM | 14709.748 | 23.766 | 3.731 |
| 3 | EGComp-AG | 287.933 | 11.683 | 3.626 | |
| 4 | SqueezeLLM | 14962.949 | 25.125 | 4.525 | |
| 4 | EGComp-AG | 310.281 | 10.812 | 4.399 |
These measurements use the same quantization configurations and checkpoints as Table R1. Both methods use native quantized inference. SqueezeLLM uses its official dense-and-sparse CUDA kernels, while EGComp-AG uses our implemented 3-bit/4-bit packed lookup-table kernels, supporting one codebook per 2048 weights and one scale per 128 weights. Both implementations compute directly from compressed weight representations without maintaining a full dequantized weight matrix for inference.
All measurements were conducted sequentially on an RTX PRO 6000 Blackwell GPU, with no other GPU compute processes, using batch size 1, 1920 input tokens, and 128 generated tokens with KV caching. We performed two warm-up runs followed by five measured runs, reporting median latency and the maximum peak allocated GPU memory. TTFT includes prefill and first-token selection; decode latency is averaged over the subsequent 127 tokens. Model loading, packing, compilation, and warm-up are excluded.
Across all eight configurations, EGComp-AG achieves lower TTFT, decode latency, and peak allocated GPU memory than the tested SqueezeLLM dense-and-sparse implementation. These results demonstrate that our codebook-and-scale representation supports native compressed inference and provide an explicit latency analysis alongside the quantization-quality results.
Response to Q3: Offline quantization time and GPU memory usage
Thank you for raising this point. In addition to inference latency, we measured the offline computation time of our framework on LLaMA-1-7B at 3 bits. Each conversion and composition stage processes all 32 Transformer layers and 224 linear weight matrices. The three stages were executed sequentially on an Intel Core i9-14900K using six single-threaded CPU workers.
| Stage | Execution device | Wall-clock time | Peak GPU memory |
|---|---|---|---|
| Fisher computation, shared | GPU + CPU accumulation | 61.26 min | 15.536 GiB |
| EGConv-A | CPU, 6 workers | 55.33 min | No GPU allocation |
| EGConv-G | CPU, 6 workers | 55.76 min | No GPU allocation |
| EGComp-AG | CPU, 6 workers | 47.06 min | No GPU allocation |
The conversion and composition times include input loading, worker startup and shutdown, optimization, and checkpoint saving. They start from existing AWQ/GPTQ reference weights and Fisher statistics, excluding source-quantizer generation, deployment packing, and evaluation. Fisher computation was profiled separately using 100 C4 sequences of 512 tokens with batch size 1; its memory figure is the peak PyTorch allocated GPU memory, including the resident model.
The three CPU stages take 158.15 minutes in total, and conversion and composition require no GPU memory. These complete-model measurements complement our inference-latency analysis by quantifying the offline preparation cost.
Thank you for the additional experiments and latency measurements. My main concern is whether composition adds enough value beyond standalone conversion to justify its added complexity.
Contribution of composition. In Table 2, most of the improvement on Qwen3-8B at 3 bits comes from standalone conversion, with composition reducing WikiText-2 perplexity only from 7.865 to 7.860.
Robustness. Table 2 reports a single result per setting without variability estimates, making it difficult to assess whether the small gains from composition are reproducible.
Comparison with stronger quantizers. The storage-matched SqueezeLLM comparison broadens the evaluation, but it remains unclear whether composition improves more recent quantizers or outperforms them at the same storage budget.
Practical efficiency. The GPU results show a speed advantage over the tested SqueezeLLM implementation, but direct latency comparisons with AWQ, GPTQ, and standalone conversion are needed to assess the trade-off between quality and inference cost.
Given the limited evidence that composition provides a reproducible and practically meaningful benefit, I am lowering my rating from Weak Reject to Reject.
Response to Q1: Coverage of models, methods, and benchmarks
Thank you for raising this point. The original submission already evaluates Qwen3-8B-Base alongside LLaMA-1, LLaMA-2, OPT, and Mistral, covering five model families. It includes 3-bit and 4-bit perplexity evaluations on WikiText-2 and C4, together with seven zero-shot downstream tasks on four models. The benefits of conversion and composition are therefore also demonstrated on Qwen3.
We additionally compared against SqueezeLLM on four models at both bit widths, using the same original weights and precomputed Fisher statistics. Average storage bit widths are matched using dense-and-sparse SqueezeLLM, with approximately 0.24%–0.31% of weights in the high-precision sparse branch. The accounting includes low-bit indices, codebooks, applicable scales, and sparse values and indexing overhead. Average bits are reported per quantized weight.
Table R1. WikiText-2 perplexity comparison under matched average storage bit widths.
EGComp-AG achieves lower perplexity in seven of eight configurations at matched average bit widths, establishing its advantage over an existing non-uniform baseline in the tested settings. These additional experiments broaden method coverage under the original benchmark protocol.
Response to Q2: Magnitude and consistency of the gains
Thank you for raising this point. The contribution is error-guided conversion and storage-preserving composition of existing quantizers. The results show improvements across model families, datasets, and bit widths:
For example, at 3 bits on Qwen3-8B-Base, WikiText-2 PPL decreases from 8.359 for the better source quantizer to 7.860, and C4 PPL decreases from 14.297 to 13.702. The composition gains are additional to those already obtained through conversion and retain exactly the same final representation and storage budget as a single converted quantizer. This combination of model-level quality improvements and zero additional composition storage is the central contribution.
Response to Q4: Experimental validation of element-wise complementarity
Thank you for raising this point. Figure 1 and Table 1 serve two connected purposes: identifying weight-level complementarity and experimentally validating its model-level value. Figure 1 shows that AWQ and GPTQ reconstruct different weight positions more accurately. We then directly test this observation through MinError, which selects the lower-error reconstruction from the two source quantizers at each weight position and is evaluated as a complete model.
In Table 1, MinError achieves lower perplexity than both AWQ and GPTQ in all 16 configurations, covering four models, two bit widths, and two datasets. This provides direct experimental validation that the observed element-wise complementarity can be exploited to improve model quality.
GPTQ's activation-dependent layer objective is compatible with another quantizer reconstructing some individual weights more accurately. Our use of element-wise error does not require it to be equivalent to GPTQ's objective: its usefulness for composition is directly tested through the resulting model's perplexity. Table 1 demonstrates that this selection criterion translates into model-level gains across all reported settings.
MinError validates the intuition through direct composition; the subsequent EGConv/EGComp framework addresses its additional storage requirements by combining source quantizers within the storage budget of a single converted quantizer.