acceptodds
Under review as a conference paper at ICLR 2027

Projected-Distortion Guided Quantization

Abstract

Post-training quantization (PTQ) reduces the memory and latency costs of large language models (LLMs), but existing methods degrade sharply at 2–3 bits, in part because codebook fitting, outlier handling, and pruning are designed and tuned separately. We present Projected-Distortion Guided Quantization (PDGQ), which drives all three with a single activation-aware measure of output distortion. PDGQ combines (1) a maximum-entropy codebook refined by activation-weighted Lloyd iterations, (2) Projected-Distortion Outlier Restoration (PDOR), which restores weights whose distortion is large relative to a rate–distortion reference, on a scale shared across layers and bit-widths, and (3) Distortion-Consistent Pruning (DCP), which zeroes weights that zero represents better than their codeword. On LLaMA and Qwen3 models, PDGQ matches recent PTQ methods in perplexity and zero-shot accuracy at 4 bits and outperforms them by a margin that grows as the bit budget shrinks, being largest at nominal 2–3 bits, with a few narrow exceptions that we report. With simple activation and KV-cache quantization, PDGQ also preserves DeepSeek-R1-Distill accuracy on GSM8K and MATH-500 under W4A4KV4 better than FlatQuant and QuaRot.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Low-Bit Quantization, Model Compression, Rate-Distortion Theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ecZdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the limitations of low-bit PTQ by treating codebook optimization, outlier restoration, and pruning as related decisions under a unified projected-distortion metric. Beyond optimizing the codebook with activation-aware weighting and entropy regularization, PDGQ uses a rate-distortion-based criterion to select weights for full-precision restoration and removes weights when zeroing introduces less error than quantization.

Critical strengths
  1. The activation-weighted Lloyd refinement explicitly accounts for input-channel importance during codebook optimization, improving low-bit quantization performance over unweighted refinement.

  2. The evaluation covers multiple model families and bit-widths, with results on both language modeling perplexity and downstream reasoning tasks.

Critical weaknesses
  1. PDOR uses a rate-distortion-based reference to normalize quantization errors and derives a shared restoration threshold based on a chi-square distribution. However, the statistic in Eq. (6) also includes a normalized activation-energy factor, whose effect on the threshold is not fully explained. Could the authors clarify the derivation and provide evidence that the threshold generalizes across layers and bit-widths?

  2. The codebook optimization combines activation-weighted Lloyd refinement with entropy regularization, while DCP simply compares the errors of quantization and zeroing. Although these designs are reasonable, it remains unclear how much they improve over straightforward alternatives. More detailed ablations against activation-weighted quantization, magnitude-based outlier restoration, and zero-aware quantization would help clarify their contributions.

  3. The projected-distortion objective in Eq. (1) is approximated by a per-weight sum in Eq. (2), ignoring the off-diagonal terms of the input covariance matrix Σx\Sigma_x. Since these terms capture correlations between input channels, how much does this approximation affect the distortion estimates and subsequent restoration decisions? A comparison with the full covariance-based objective on representative layers would help clarify this point.

  4. PDGQ restores some weights to full precision and introduces additional indexing or masking overhead, so its effective bit-width can be higher than the nominal quantization bit-width. Comparing methods at matched effective bit-width, including these overheads, would provide a clearer picture of its actual compression–accuracy trade-off.

Questions for authors
  1. Could the authors clarify how sensitive PDGQ is to the choice of the entropy regularization coefficient, and whether the observed gains remain consistent across different models and bit-widths?

  2. How much does the diagonal approximation in Eq. (2) affect the distortion estimates and restoration decisions compared with using the full covariance matrix?

  3. Could the authors report comparisons at matched effective bit-widths, including the overhead of storing restored weights and their indices, to clarify whether PDGQ retains its advantage under the same storage budget?

Reviewer_hum9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PDGQ, a post-training quantization method that combines activation-weighted codebook optimization, selective restoration of high-error weights, and distortion-based pruning. The reported results are promising, particularly at nominal 2–3 bits.

Critical strengths

The method is relatively simple, and using a common distortion measure across its components is well motivated. Evaluation covers several model families and sizes, while the component studies demonstrate useful improvements from activation weighting and restoration. I also appreciate the empirical checks of the diagonal approximation in Appendix D.2.

Critical weaknesses

My main concerns are:

  1. The storage accounting appears incomplete. Section 4.1 assigns a separate codebook to every 128-weight group, but Eq. (54) does not include codebook storage. Assuming FP16 codewords, this adds 0.5, 1, and 2 bits per weight at nominal 2, 3, and 4 bits, respectively. For example, Table 10’s nominal 2-bit configuration would require approximately 3.36 rather than 2.86 bits per weight. Comparisons at matched total storage are needed to establish the compression advantage.
  2. The restoration comparison uses unequal budgets. In Table 13, PDOR restores 0.93% of weights versus 0.26% for the magnitude rule on LLaMA-2-7B at 2 bits. Although acknowledged, this prevents the comparison from isolating selection quality. The simplified comparator also does not cover the sensitive-weight restoration used by SqueezeLLM. Comparisons against magnitude and raw projected-error ranking at the same restored fraction would be more informative.
  3. The entropy regularizer needs a separate ablation. Table 10 examines activation weighting, while Figure 2 examines initialization. Neither establishes the benefit of the entropy penalty itself. A comparison with γ=0\gamma=0, holding initialization and the remaining pipeline fixed, would clarify this contribution. Could the authors provide complete serialized model sizes, comparisons at matched restoration budgets, and an isolated entropy-regularization ablation?
Questions for authors

See above

Reviewer_uwRHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PDGQ, a posttraining scalar quantization method that uses activation aware distortion metric: the expected squared layer output error under a diagonal activation covariance approximation

  • For each 128 weight row group, it initializes a nonuniform scalar codebook of 2^b centroids using empirical quantiles and refines them via Lloyd iterations with entropy penalty on codeword usage
  • Weights with weighted quantization error large relative to the group Gaussian rate distortion noise floor are saved in full FP16 precision using a \chi^2 cutoff.
  • the weights that are not saved in FP and lie closer to zero than to their assigned codeword are set to zero

The paper evaluates the proposed method on LLaMA and Qwen3 models on perplexity and zero-shot benchmarks, as well as DeepSeek-R1-Distill-Qwen on math reasoning. Ablations show that each stage reduces perplexity within the pipeline.

Critical strengths
  • The ablations in tables 10,11 show that each stage of PDGQ noticeably improves perplexity. Replacing unweighted Lloyd refinement with activation weighted Lloyd updates (at the exact same quantile initialization and 2-bit codebook size, with zero extra storage) drops LLaMA2-7B WikiText2 perplexity from 10.58 to 8.68. Adding PDOR and DCP further reduces perplexity to 7.55 and 6.50 on LLaMA-2-7B, and shows similar gains on Qwen3-8B-Base
  • The method requires only calibration statistics and Lloyd iterations without backpropagation.
Critical weaknesses
  • In Tables 1,2 PDGQ is compared against 2,3,4 bit baselines using the codebook bit width b, although PDGQ stores additional FP16 outliers and pruning mask. In App F the authors compute an effective bit-width that adds (some of) the additional costs and brings the number of bits on LLaMA-2-7B to 2.86, 3.64, and 4.32 bits per weight respectively. However, this is also incomplete because it omits the cost of storing the learned codebooks. The method fits distinct codebooks (c1,...,cK)(c_1, ..., c_K) of K=2bK =2^b centroids for every row group of 128 weights. If each centroid is stored in s bits, this adds: s2b128\frac{s 2^b}{128} bits per weight. With FP16 centroids, the codebook alone adds 0.50 bits/weight for b=2, and 1 bit/weight at b=3, and 2 bits/weight for b=4. So, 2bit PDGQ actually uses ~3.36 bits per weight; and performs worse that some existing methods which use 3 bits.
  • Table 13 compares PDOR against a fixed 3σ3\sigma magnitude rule while restoring 2.4x to 5.7x more FP weights. This uses a much larger outlier budget than the baselines and makes the comaprison unfair. The codeword entropy penalty during Lloyd refinement is claimed as a core contribution, but I didn't find an ablation against a zero penalty baseline.
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.