acceptodds
Under review as a conference paper at ICLR 2027

Error-Guided Refinement and Composition for LLM Quantization

Abstract

Post-training quantization has emerged as an active research area for enabling the efficient deployment of large language models (LLMs). A wide range of quantization methods have been proposed, each based on different design intuitions. We observe that one method can have smaller quantization errors on some weights, while another has smaller errors on others. These differences can affect model performance and may reflect how each method prioritizes different weights. Motivated by this observation, we develop an error-guided approach for both improving individual quantizers and combining their strengths. We first show that selecting more accurate weight approximations from different methods can improve model quality, but introduces additional storage to record the selected method for each weight. To address this issue, we propose an error-guided conversion method that constructs non-uniform variants of existing quantizers using their quantization errors. This conversion provides a common codebook-based representation for different methods. We then combine their codebooks into a single codebook with the same number of entries as an individual non-uniform quantizer, avoiding additional per-weight method labels. Experiments on multiple LLMs show that the proposed conversion can improve perplexity over the original uniform quantizers, while codebook composition provides further gains in several evaluated settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sS2wOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors explore a novel approach that combines weights quantized by two different quantization methods, AWQ and GPTQ, to obtain a single quantized model, and they report performance improvements from this approach.

Critical strengths

To the best of my knowledge, the idea of combining the results of two different quantization methods has not been explored before. The main strength of this paper is to propose such novel idea and verfying it with experiments.

Critical weaknesses

The idea of the paper is interesting, but the following aspects need to be improved.

  1. Lack of experiments with recent models, methods, and benchmarks.
  2. Marginal performance gain
  3. Absence of latency analysis.
  4. The authors use element-wise weight reconstruction errors ∣W−W^∣|W-\widehat{W}| for defining the fraction of weights for which each method is more accurate in Figure 1, while GPTQ's objective function ∣∣WX−W^X∣∣||WX-\widehat{W}X|| includes the distribution of activations.
Questions for authors

Could you provide your responses to the points raised in the Weakness section?

4 comments

Response to Q1: Coverage of models, methods, and benchmarks

Thank you for raising this point. The original submission already evaluates Qwen3-8B-Base alongside LLaMA-1, LLaMA-2, OPT, and Mistral, covering five model families. It includes 3-bit and 4-bit perplexity evaluations on WikiText-2 and C4, together with seven zero-shot downstream tasks on four models. The benefits of conversion and composition are therefore also demonstrated on Qwen3.

We additionally compared against SqueezeLLM on four models at both bit widths, using the same original weights and precomputed Fisher statistics. Average storage bit widths are matched using dense-and-sparse SqueezeLLM, with approximately 0.24%–0.31% of weights in the high-precision sparse branch. The accounting includes low-bit indices, codebooks, applicable scales, and sparse values and indexing overhead. Average bits are reported per quantized weight.

Table R1. WikiText-2 perplexity comparison under matched average storage bit widths.

ModelBitsSqueezeLLM Avg. bitsSqueezeLLM Wiki PPL ↓EGComp-AG Avg. bitsEGComp-AG Wiki PPL ↓
LLaMA-1-7B33.1891196.1983.1891196.162
LLaMA-1-7B44.2532385.8014.2532385.748
OPT-6.7B33.187511.3133.187511.372
OPT-6.7B44.2510.9984.2510.865
Qwen3-8B-Base33.18757.9053.18757.860
Qwen3-8B-Base44.257.1754.257.167
Mistral-7B-v0.133.18755.7373.18755.729
Mistral-7B-v0.144.255.3504.255.349

EGComp-AG achieves lower perplexity in seven of eight configurations at matched average bit widths, establishing its advantage over an existing non-uniform baseline in the tested settings. These additional experiments broaden method coverage under the original benchmark protocol.


Response to Q2: Magnitude and consistency of the gains

Thank you for raising this point. The contribution is error-guided conversion and storage-preserving composition of existing quantizers. The results show improvements across model families, datasets, and bit widths:

  • In Table 2, EGComp-AG outperforms both original AWQ and GPTQ in all 20 configurations.
  • It also outperforms the better converted endpoint in 17 of the 20 configurations, with no additional storage over an individual converted quantizer.
  • In Table 4, average accuracy across seven zero-shot tasks improves over the better original source quantizer on all four models.

For example, at 3 bits on Qwen3-8B-Base, WikiText-2 PPL decreases from 8.359 for the better source quantizer to 7.860, and C4 PPL decreases from 14.297 to 13.702. The composition gains are additional to those already obtained through conversion and retain exactly the same final representation and storage budget as a single converted quantizer. This combination of model-level quality improvements and zero additional composition storage is the central contribution.


Response to Q4: Experimental validation of element-wise complementarity

Thank you for raising this point. Figure 1 and Table 1 serve two connected purposes: identifying weight-level complementarity and experimentally validating its model-level value. Figure 1 shows that AWQ and GPTQ reconstruct different weight positions more accurately. We then directly test this observation through MinError, which selects the lower-error reconstruction from the two source quantizers at each weight position and is evaluated as a complete model.

In Table 1, MinError achieves lower perplexity than both AWQ and GPTQ in all 16 configurations, covering four models, two bit widths, and two datasets. This provides direct experimental validation that the observed element-wise complementarity can be exploited to improve model quality.

GPTQ's activation-dependent layer objective is compatible with another quantizer reconstructing some individual weights more accurately. Our use of element-wise error does not require it to be equivalent to GPTQ's objective: its usefulness for composition is directly tested through the resulting model's perplexity. Table 1 demonstrates that this selection criterion translates into model-level gains across all reported settings.

MinError validates the intuition through direct composition; the subsequent EGConv/EGComp framework addresses its additional storage requirements by combining source quantizers within the storage budget of a single converted quantizer.

Response to Q3: Native quantized inference latency and GPU memory

Thank you for highlighting the need for latency analysis. We have implemented packed 3-bit and 4-bit lookup-table kernels for EGComp-AG and added native quantized inference measurements across four models at both bit widths.

Table R3. Native quantized inference latency and peak GPU memory for the quantization configurations in Table R1.

ModelBitsMethodTTFT (ms) ↓Decode (ms/token) ↓Peak GPU memory (GiB) ↓
LLaMA-1-7B3SqueezeLLM19631.94230.7804.110
3EGComp-AG258.91111.1654.028
4SqueezeLLM19682.45631.6624.973
4EGComp-AG286.86910.3554.874
OPT-6.7B3SqueezeLLM15774.17121.1634.820
3EGComp-AG246.9999.8014.802
4SqueezeLLM16080.23921.7835.634
4EGComp-AG269.5679.1795.599
Qwen3-8B-Base3SqueezeLLM14293.97824.7665.473
3EGComp-AG294.07014.2835.382
4SqueezeLLM15065.18326.6216.342
4EGComp-AG316.91313.9436.228
Mistral-7B-v0.13SqueezeLLM14709.74823.7663.731
3EGComp-AG287.93311.6833.626
4SqueezeLLM14962.94925.1254.525
4EGComp-AG310.28110.8124.399

These measurements use the same quantization configurations and checkpoints as Table R1. Both methods use native quantized inference. SqueezeLLM uses its official dense-and-sparse CUDA kernels, while EGComp-AG uses our implemented 3-bit/4-bit packed lookup-table kernels, supporting one codebook per 2048 weights and one scale per 128 weights. Both implementations compute directly from compressed weight representations without maintaining a full dequantized weight matrix for inference.

All measurements were conducted sequentially on an RTX PRO 6000 Blackwell GPU, with no other GPU compute processes, using batch size 1, 1920 input tokens, and 128 generated tokens with KV caching. We performed two warm-up runs followed by five measured runs, reporting median latency and the maximum peak allocated GPU memory. TTFT includes prefill and first-token selection; decode latency is averaged over the subsequent 127 tokens. Model loading, packing, compilation, and warm-up are excluded.

Across all eight configurations, EGComp-AG achieves lower TTFT, decode latency, and peak allocated GPU memory than the tested SqueezeLLM dense-and-sparse implementation. These results demonstrate that our codebook-and-scale representation supports native compressed inference and provide an explicit latency analysis alongside the quantization-quality results.

Response to Q3: Offline quantization time and GPU memory usage

Thank you for raising this point. In addition to inference latency, we measured the offline computation time of our framework on LLaMA-1-7B at 3 bits. Each conversion and composition stage processes all 32 Transformer layers and 224 linear weight matrices. The three stages were executed sequentially on an Intel Core i9-14900K using six single-threaded CPU workers.

StageExecution deviceWall-clock timePeak GPU memory
Fisher computation, sharedGPU + CPU accumulation61.26 min15.536 GiB
EGConv-ACPU, 6 workers55.33 minNo GPU allocation
EGConv-GCPU, 6 workers55.76 minNo GPU allocation
EGComp-AGCPU, 6 workers47.06 minNo GPU allocation

The conversion and composition times include input loading, worker startup and shutdown, optimization, and checkpoint saving. They start from existing AWQ/GPTQ reference weights and Fisher statistics, excluding source-quantizer generation, deployment packing, and evaluation. Fisher computation was profiled separately using 100 C4 sequences of 512 tokens with batch size 1; its memory figure is the peak PyTorch allocated GPU memory, including the resident model.

The three CPU stages take 158.15 minutes in total, and conversion and composition require no GPU memory. These complete-model measurements complement our inference-latency analysis by quantifying the offline preparation cost.

Thank you for the additional experiments and latency measurements. My main concern is whether composition adds enough value beyond standalone conversion to justify its added complexity.

Contribution of composition. In Table 2, most of the improvement on Qwen3-8B at 3 bits comes from standalone conversion, with composition reducing WikiText-2 perplexity only from 7.865 to 7.860.

Robustness. Table 2 reports a single result per setting without variability estimates, making it difficult to assess whether the small gains from composition are reproducible.

Comparison with stronger quantizers. The storage-matched SqueezeLLM comparison broadens the evaluation, but it remains unclear whether composition improves more recent quantizers or outperforms them at the same storage budget.

Practical efficiency. The GPU results show a speed advantage over the tested SqueezeLLM implementation, but direct latency comparisons with AWQ, GPTQ, and standalone conversion are needed to assess the trade-off between quality and inference cost.

Given the limited evidence that composition provides a reproducible and practically meaningful benefit, I am lowering my rating from Weak Reject to Reject.

Reviewer_n6fHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper starts from the observation that different weight-only post-training quantizers (AWQ, GPTQ, RTN) have smaller element-wise errors on different weights. It first shows that choosing the lower-error reconstruction per weight ("MinError") lowers perplexity, at a cost of 1.25 extra bits per weight for a selection mask and the second method's parameters. To avoid that overhead, it re-quantizes each source method into a common non-uniform format: b-bit indices, an FP16 codebook per 2048-weight group, and an FP16 scale per 128-weight subgroup. The codebooks and scales are fit by alternating weighted k-means on the original weights.

Critical strengths
  • MinError needs a 1-bit mask and two parameter sets (5.50 vs 4.25 bits per weight at 4 bits; Table 8). Mapping both methods into one codebook format lets the merged model keep a single index per weight at the storage of one quantizer.
  • It spans five model families, two bit widths, two perplexity datasets, all three pairwise compositions of AWQ, GPTQ and RTN (Fig. 2, Fig. 3), and zero-shot tasks. The tables are internally consistent.
  • Conversion to the non-uniform format consistently improves on the uniform source quantizers at equal nominal storage. This holds in all 40 source/setting pairs of Table 2, by 0.04–0.63 WikiText-2 PPL at 3 bits, which is a useful practical data point.
Critical weaknesses
  • The central point of the paper is that error-guided conversion keeps what each source quantizer does well, and that composition combines these strengths. The paper's own ablation points the other way.
  • Together these suggest that the improvements in Table 2 over AWQ and GPTQ (up to 0.63 PPL) come from moving to a Fisher-weighted non-uniform codebook format, which is the SqueezeLLM recipe (Kim et al., 2024) with added group scales. That format also uses gradient information that AWQ and GPTQ do not use. The source quantizer's errors seem to contribute little.
  • To support "combining strengths", the paper should compare EGComp with EGConv-None given the same total iteration budget, with several restarts or a stronger initialization (e.g. k-means++, or the best of two inits by objective value). It should also report the final Fisher-weighted objective values next to the perplexities.
  • SqueezeLLM is cited for the Fisher-weighted clustering the method relies on (l. 136–137, 245–247, 262–263) but is not compared. Its published WikiText-2 numbers (Kim et al., https://arxiv.org/abs/2306.07629, Table 1 and App. D) are close to EGComp-AG. Calibration data differ, so these numbers are only indicative. A controlled comparison under the paper's protocol is needed before the method can be said to add anything over SqueezeLLM.
Questions for authors
  • Can you report EGConv-None for all settings in Table 2 and Figs. 2–3? Can you also compare EGComp-AG with EGConv-None run for the same total number of alternating iterations and with several random or k-means++ restarts, reporting the final Fisher-weighted objective?
  • How does EGComp-AG compare with SqueezeLLM (dense-only, and with its codebook granularity matched to your storage budget), and with sequential AWQ→GPTQ, under your calibration and evaluation protocol?
  • How long do the Fisher computation, the conversion and the composition take for a 7B model, and how much GPU memory do they need? Do you have a lookup-table kernel for the per-2048-group codebook plus per-128 scale format, and how does its latency compare with standard INT4/INT3 kernels
5 comments

Response to Q1: EGConv-None coverage, fixed-60-iteration controls, and final Fisher objectives

Thank you for this suggestion. We completed EGConv-None evaluations for all model/bit-width/dataset settings covered by Table 2 and Figures 2–3: five models at 3 and 4 bits on WikiText-2 and C4, totaling 20 configurations. EGConv-None uses only the Fisher coefficients FijF_{ij}, without source-quantization error guidance. Its codebook grouping, scale grouping, and parameter precision match the corresponding EGConv/EGComp settings. The lowest PPL for each model and column is highlighted in bold.

ModelMethod3-bit Wiki ↓3-bit C4 ↓4-bit Wiki ↓4-bit C4 ↓
LLaMA-1-7BEGConv-None6.1770668.7900215.7741058.212275
EGConv-A6.1634288.7792125.7694228.206393
EGConv-G6.1756628.7817155.7816978.211916
EGComp-AG6.1621278.7758885.7484238.208260
LLaMA-2-7BEGConv-None6.0145338.6150825.5782658.013047
EGConv-A5.9976848.6104355.5787388.016837
EGConv-G6.0183148.6046035.5813178.014694
EGComp-AG5.9869518.5970715.5725918.007054
OPT-6.7BEGConv-None11.44787113.63707310.88820513.213242
EGConv-A11.43575613.63646110.87966413.213053
EGConv-G11.34424913.63381210.90847813.208385
EGComp-AG11.37151813.61936810.86498513.202644
Qwen3-8B-BaseEGConv-None7.90475613.7513957.16630412.666818
EGConv-A7.91036513.7579307.17632012.675772
EGConv-G7.86473213.7195477.16600512.674183
EGComp-AG7.85989313.7023337.16727412.663605
Mistral-7B-v0.1EGConv-None5.7728819.7997115.3556569.143740
EGConv-A5.7352819.7965355.3610549.142859
EGConv-G5.7386229.8036215.3533479.140527
EGComp-AG5.7293439.7885595.3485789.136616

EGComp-AG achieves lower perplexity than EGConv-None in 19 of the 20 configurations.

We additionally conducted fixed-60-iteration experiments on four models at 3 bits. For each model, we independently initialized Fisher-weighted K-means++ with seeds 0, 1, and 2. Every quantization group received exactly 60 alternating updates of centers, scales, and assignments, without early stopping. We report each complete-model run separately, without selecting or combining results across seeds.

Alongside WikiText-2 and C4 perplexity, we report the final Fisher-weighted objective, summed over all quantized weight matrices:

J=∑ijFij(Wij−W^ij)2.J=\sum_{ij}F_{ij}(W_{ij}-\widehat{W}_{ij})^2.

Within each model, all methods use identical original weights and Fisher coefficients for this evaluation.

ModelMethodFinal Fisher objective ↓Wiki PPL ↓C4 PPL ↓
LLaMA-1-7BNone-60, seed=02.230074e-026.1746208.786635
None-60, seed=12.228957e-026.1838878.800527
None-60, seed=22.230567e-026.2075848.791046
EGComp-AG2.194560e-026.1621278.775888
OPT-6.7BNone-60, seed=01.925226e-0211.44480413.639703
None-60, seed=11.924286e-0211.46143113.641225
None-60, seed=21.924565e-0211.39206013.638968
EGComp-AG1.889153e-0211.37151813.619368
Qwen3-8B-BaseNone-60, seed=03.896097e-047.90727813.748894
None-60, seed=13.893222e-047.90142313.695406
None-60, seed=23.898591e-047.87475913.738326
EGComp-AG3.799411e-047.85989313.702333
Mistral-7B-v0.1None-60, seed=03.112791e-045.7727839.801759
None-60, seed=13.111339e-045.7610359.832530
None-60, seed=23.113859e-045.7537359.829506
EGComp-AG3.041128e-045.7293439.788559

EGComp-AG achieves a lower final Fisher objective in all 12 comparisons against the independent None-60 runs and lower perplexity in 23 of the 24 seed-wise comparisons. Thus, in these fixed-60-iteration experiments, extending Fisher-only optimization and varying its initialization does not generally reproduce the results achieved by composition.

Response to Q2: Comparisons with SqueezeLLM and sequential AWQ→GPTQ

Comparison with SqueezeLLM. We conducted additional comparisons on four models at both 3 and 4 bits, using the same original model weights and precomputed Fisher statistics. All configurations were evaluated on WikiText-2 with the same tokenization and sequence length.

To match storage budgets, we used a dense-and-sparse SqueezeLLM configuration, allocating approximately 0.24%–0.31% of weights to the high-precision sparse branch. Average bits per quantized weight account for low-bit indices, codebooks, and scales, together with sparse values and sparse indexing overhead for SqueezeLLM.

Table R1. WikiText-2 perplexity comparison under matched average storage bit widths.

ModelNominal bitsSqueezeLLM Avg. bitsSqueezeLLM Wiki PPL ↓EGComp-AG Avg. bitsEGComp-AG Wiki PPL ↓
LLaMA-1-7B3-bit3.1891196.1983.1891196.162
LLaMA-1-7B4-bit4.2532385.8014.2532385.748
OPT-6.7B3-bit3.187511.3133.187511.372
OPT-6.7B4-bit4.2510.9984.2510.865
Qwen3-8B-Base3-bit3.18757.9053.18757.860
Qwen3-8B-Base4-bit4.257.1754.257.167
Mistral-7B-v0.13-bit3.18755.7373.18755.729
Mistral-7B-v0.14-bit4.255.3504.255.349

At matched average storage bit widths, EGComp-AG achieves lower perplexity than SqueezeLLM in seven of the eight configurations. These results demonstrate its advantage over an established non-uniform quantization baseline in the tested settings.

Comparison with sequential AWQ→GPTQ. We additionally evaluated a sequential baseline on four models at 3 bits. We first apply AWQ channel scaling to the floating-point model, then recollect activation statistics on the transformed model and perform GPTQ. The sequential baseline is evaluated using the paper's WikiText-2 evaluation protocol.

ModelGPTQ ↓Sequential AWQ→GPTQ ↓EGComp-AG ↓
LLaMA-1-7B6.5866.4866.162
OPT-6.7B11.83011.44911.372
Qwen3-8B-Base8.3598.4057.860
Mistral-7B-v0.16.1975.9295.729

Sequential AWQ→GPTQ improves perplexity over the GPTQ results on three of the four models. EGComp-AG achieves lower perplexity than the sequential baseline on all four models.

Together, these comparisons demonstrate EGComp-AG's advantage over a storage-matched non-uniform baseline in seven of eight configurations and over sequential AWQ→GPTQ on all four tested models. Our framework combines complementary source quantizers in a single representation and achieves these gains without increasing storage relative to an individual converted quantizer.

Reviewer_n6fHOpenReview ↗

Satisfied with the rebuttal

I thank the authors for their detailed rebuttal and additional experiments, which substantially address my original concerns. The fixed-iteration comparisons and Fisher-weighted objectives provide stronger evidence for the benefits of composition, while the SqueezeLLM comparisons strengthen the empirical contribution. However, the improvements over SqueezeLLM remain modest in several settings, raising questions about practical significance and novelty. Furthermore, the computational overhead, GPU memory requirements, and inference latency remain unaddressed, despite their importance for efficient LLM deployment. Overall, the rebuttal improves my assessment of the technical soundness, but concerns about incremental novelty and practical efficiency remain. I increase my rating from weak rejection to weak acceptance.

Reviewer_5u3POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the observation that different existing quantization methods can accurately approximate different weights and proposes an approach that exploits this complementarity. First, it demonstrates that a straightforward combination that selects the approximation with the smaller error for each weight improves model accuracy. Next, it proposes EGConv, which converts the original quantizers into a common non-uniform quantization format using their quantization errors and Fisher-based sensitivity. Furthermore, EGComp selects candidates from the two converted codebooks and reoptimizes them together with a shared scale, enabling the combination of quantizers with the same storage cost as a single converted quantizer. Across multiple models and bit widths, the paper reports lower perplexity than the original AWQ/GPTQ methods and improvements in average accuracy across seven zero-shot tasks.

Critical strengths
  1. The final EGComp method uses the same non-uniform quantization storage format as the individual EGConv-A/G methods, allowing quantizers to be combined without increasing storage requirements. While a straightforward approach that selects the original quantization method for each weight requires additional information to record the selections, the proposed method addresses this issue through conversion to a common format, codebook selection, and reoptimization.

  2. The paper demonstrates improvements over the original methods across multiple models and configurations. EGComp-AG outperforms both the original AWQ and GPTQ in all 20 configurations, and in 17 configurations, it also outperforms the better of the two converted quantizers.

Critical weaknesses
  1. The lack of comparisons with existing non-uniform quantization methods makes it difficult to assess the position of the proposed contribution. Although improvements over the original AWQ/GPTQ methods are demonstrated, the paper does not show what advantages the proposed method offers over existing non-uniform quantization methods, such as SqueezeLLM, under the same storage budget.

  2. EGConv-None also outperforms AWQ/GPTQ, indicating that non-uniform quantization and Fisher-based optimization alone can improve upon the original methods. Although Table 3 compares the results with and without error guidance, the effects of the objective function change and additional optimization in EGComp are not isolated from the effect of codebook integration. Therefore, the magnitude of the improvement attributable specifically to integration remains unclear.

Questions for authors
  1. How is the proposed method positioned relative to existing non-uniform quantization methods, such as SqueezeLLM, and does it offer any advantages under the same storage budget?

  2. What is the effect of codebook integration itself? In EGComp, in addition to integration, the coefficient of the reconstruction error is changed from Fij(1+dij)F_{ij}(1+d_{ij}) to FijF_{ij}, and additional optimization is performed. Therefore, the individual contribution of integration is unclear. If EGComp were compared against individual EGConv-A/G quantizers that are further optimized using an objective function weighted only by Fisher coefficients, without integration, would EGComp still maintain its improvements?

  3. If EGConv-None and the individual EGConv quantizers were also included in the downstream-task evaluations, would the improvements in Table 4 still be attributable to error guidance and integration?

5 comments

Response to Q1: Comparison with SqueezeLLM under matched storage budgets

Thank you for suggesting this comparison. We conducted additional comparisons with SqueezeLLM on four models at both 3 and 4 bits, using the same original model weights and precomputed Fisher statistics. All configurations were evaluated on WikiText-2 with the same tokenization and sequence length. To match the storage budgets, we used a dense-and-sparse SqueezeLLM configuration, with approximately 0.24%–0.31% of weights assigned to the high-precision sparse branch. We report the average bits per quantized weight, accounting for low-bit indices, codebooks, and scales where applicable, as well as sparse values and sparse indexing overhead for SqueezeLLM. As shown in the table, EGComp-AG achieves lower perplexity in seven of the eight configurations at matched average bit widths.

Table R1. WikiText-2 perplexity comparison under matched average storage bit widths.

ModelNominal bitsSqueezeLLM Avg. bitsSqueezeLLM Wiki PPL ↓EGComp-AG Avg. bitsEGComp-AG Wiki PPL ↓
LLaMA-1-7B3-bit3.1891196.1983.1891196.162
LLaMA-1-7B4-bit4.2532385.8014.2532385.748
OPT-6.7B3-bit3.187511.3133.187511.372
OPT-6.7B4-bit4.2510.9984.2510.865
Qwen3-8B-Base3-bit3.18757.9053.18757.860
Qwen3-8B-Base4-bit4.257.1754.257.167
Mistral-7B-v0.13-bit3.18755.7373.18755.729
Mistral-7B-v0.14-bit4.255.3504.255.349

The central question of our original submission was whether the complementary errors of existing AWQ and GPTQ quantizers could be exploited through error-guided conversion and composition, while keeping the composed quantizer within the storage budget of a single converted quantizer. Accordingly, our original experiments focused on AWQ, GPTQ, and their converted counterparts to assess conversion and composition separately.

Response to Q2: Effect of codebook integration

Thank you for raising this question. Following your suggestion, we conducted an additional 3-bit control experiment on four models. Starting separately from the existing EGConv-A and EGConv-G quantizers, we replaced the optimization weights Fij(1+dij)F_{ij}(1+d_{ij}) with FijF_{ij} alone and further optimized the codebook centers, scales, and assignments for up to 30 additional iterations, using the same iteration limit and stopping criterion as the original EGComp-AG composition stage. Each control retains its own single-source codebook, without introducing candidate centers from another source or performing codebook integration.

The following table reports 3-bit perplexity on WikiText-2 and C4. “+ Fisher” denotes the additional Fisher-only refinement described above. The lowest PPL in each row is highlighted in bold.

ModelDatasetEGConv-A + Fisher ↓EGConv-G + Fisher ↓EGComp-AG ↓
LLaMA-2-7BWikiText-25.9966636.0208235.986951
LLaMA-2-7BC48.6182338.6075688.597071
OPT-6.7BWikiText-211.46661811.35258911.371518
OPT-6.7BC413.63505413.63265113.619368
Qwen3-8B-BaseWikiText-27.9001687.8648827.859893
Qwen3-8B-BaseC413.74851613.71924313.702333
Mistral-7B-v0.1WikiText-25.7400705.7394105.729343
Mistral-7B-v0.1C49.8066489.8073459.788559

EGComp-AG achieves lower perplexity than both Fisher-refined single-source quantizers in seven of the eight configurations across four models and two datasets. This control demonstrates that switching to Fisher-only weighting and performing additional refinement alone does not generally reproduce the gains achieved by composition, providing direct evidence for the additional benefit of codebook integration.

Response to Q3: Additional single-source controls and downstream-task evaluations

Thank you for suggesting these controls. We completed 3-bit evaluations of EGConv-None, EGConv-A, and EGConv-G on five models across all seven zero-shot tasks. We additionally evaluated EGComp-AGN, which combines these three sources. Within each model, all methods use the same final codebook grouping, scale grouping, and parameter precision.

ModelEGConv-NoneEGConv-AEGConv-GEGComp-AGN
LLaMA-1-7B63.56963.03363.12563.991
LLaMA-2-7B64.10464.38864.52364.299
OPT-6.7B57.74157.58857.57957.864
Qwen3-8B-Base67.86267.93368.60068.608
Mistral-7B-v0.169.73169.73169.51269.891

Values are average accuracy (%) across seven tasks; higher is better. Bold indicates the best result within each model.

The EGComp-AGN reported here is a supplementary three-source composition variant. During candidate selection, each trial deletion is scored after one shared-scale adaptation; subsequent refinement follows the scale → assignment → center update order.

EGComp-AGN outperforms EGConv-None on all five models and the strongest individual converted quantizer on four of the five models. Relative to EGConv-None, average accuracy increases by 0.422, 0.195, 0.123, 0.746, and 0.160 percentage points, respectively. Thus, with the additional single-source controls included, the composition procedure still demonstrates downstream improvements beyond Fisher-only conversion and strong individual converted endpoints.

Meanwhile, EGComp-AGN reduces the 24 candidate centers from three sources to eight final centers, retaining one codebook per 2048 weights, one FP16 scale per 128 weights, and one 3-bit index per weight. No source-selection labels are stored, and the final representation requires no additional storage relative to an individual converted quantizer.

These results support the central value of our composition framework: combining multiple converted sources can further improve downstream accuracy while preserving the storage budget of a single converted quantizer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.