acceptodds
Withdrawn from ICLR 2027

PATCH: Propagation-Aware Two-Stage Quantization Error Compensation via Fine-Grained Correction for High-Fidelity 2-Bit LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Extreme low-bit quantization reduces the memory and data-movement costs of large language models (LLMs), but 2-bit precision introduces substantial errors that accumulate across layers. Existing compensation methods are constrained by both correction expressiveness and optimization scope: limited-rank corrections struggle to capture the error structure, channel-wise scaling overlooks intra-channel heterogeneity, and locally optimal compensation strengths need not minimize end-to-end loss. We introduce PATCH, a propagation-aware two-stage framework that decouples fine-grained correction from layer-wise strength regulation. The first stage fits group-wise compensation using activations from the quantized network, capturing heterogeneous intra-channel errors without an explicit rank constraint. The second stage fixes the learned corrections and jointly optimizes layer-wise regulation coefficients against an end-to-end objective, accounting for error propagation while preserving the correction directions. The resulting coefficients are fused into existing quantization parameters, avoiding a separate compensation branch. Across Gemma-2, Llama-3, and Qwen3 models of varying scales under 2-bit GPTQ, PATCH reduces WikiText-2 perplexity by 24.7% on average and improves average zero-shot accuracy by 2.03 percentage points relative to the best competing compensation method for each model and metric. These results demonstrate that jointly addressing correction expressiveness and network-wide compensation strength enables effective accuracy recovery in extreme low-bit LLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Model, Quantization, Post-training quantization, Quantization error compensation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2Zq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a coherent two-stage design that explicitly separates fine-grained error representation from layer-wise compensation-strength selection. The empirical evaluation is relatively broad across three model families and three PTQ methods.

Critical strengths

The main experiments include six models spanning Llama, Qwen, and Gemma and evaluate GPTQ, AWQ, and QuIP under 2-bit weight-only quantization, WikiText-2 PPL results show that PATCH improves substantially over the uncompensated GPTQ and other compensation methods. The ablation experiments also show the contribution of two stages, demonstrating the necessity of the second stage.

Critical weaknesses

Under the w2a16 precision setting, although the PPL after error compensation show substantial improvement over other methods, they still fall short of QAT-based methods such as EfficientQAT. Under the W2A16 setting, post-training quantization (PTQ) seems to offer limited value.

Questions for authors
  1. In the isolated setting of Figure 2(b), should the end-to-end loss curve reach 1 at its optimum? Please supplement the definitions of local error and end-to-end loss in Figure 2.
  2. Stage one optimizes m_{c,g}, while stage two optimizes the magnitude of m_{c,g} within each block. Why not directly train with a joint loss in a single stage, rather than splitting the procedure into two stages?
  3. Why does stage two remain at block-level granularity instead of refining the granularity down to the group level as well?
Reviewer_wVfZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents PATCH, a two-stage compensation framework for 2-bit weight-only LLM quantization. It first applies fine-grained group-wise vector compensation to better capture heterogeneous quantization errors, then performs layer-wise end-to-end strength regulation to account for error propagation across the network.

Critical strengths
  • The paper clearly separates the limitations of existing compensation methods into representation capacity and local-to-global objective mismatch.
  • PATCH substantially improves error correction while fusing the compensation coefficients into existing quantization scales, avoiding a separate correction branch during inference.
  • The paper evaluates end-to-end latency on both a server GPU and an on-device NPU, providing useful evidence from different computing platforms.
Critical weaknesses
  • Weak motivation: Fig. 1(a) does not convincingly demonstrate that low-rank compensation is insufficient, as even a small rank recovers 20–30% of the error energy. Likewise, the residual variation in Fig. 1(b) appears modest. In Fig. 2, how are the local error and end-to-end loss defined? Also, if the optimal compensation strength is often far from 1, does this suggest that the compensation itself has limited utility or is highly task-dependent?
  • Limited analysis of vector compensation: PATCH builds on vector compensation, which corrects quantization error mainly through scaling, but the paper does not sufficiently explain why this parameterization should outperform low-rank compensation.
  • Potential calibration overfitting: Both proposed stages rely on calibration-based training. The robustness to calibration domain and generalization to diverse downstream tasks remain unclear.
  • Limited algorithmic novelty: Stage 1 is largely a group-wise extension of channel-wise vector compensation, while Stage 2 mainly changes the optimization scope from local to global. Thus, the overall algorithmic contribution appears incremental.
  • Unclear practical utility of 2-bit models: Despite the relative improvements, the performance remains substantially below BF16, raising questions about whether the resulting 2-bit models are practically useful.
  • Baseline fairness and hardware results: How does PATCH compare with other state-of-the-art 2-bit weight-only quantization methods? Since PATCH specifically targets weight-only quantization, a direct comparison with methods such as QuIP#[1], AQLM[2], and QTIP[3] would better clarify its competitiveness. In addition, SERQ reports unexpectedly high latency despite being designed to improve LQER’s hardware efficiency, which makes the reported hardware comparison difficult to assess.

 


[1] A. Tseng et al., "QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks", ICML, 2024.
[2] V. Egiazarian et al., "Extreme Compression of Large Language Models via Additive Quantization", ICML, 2024.
[3] A. Tseng et al., "QTIP: Quantization with Trellises and Incoherence Processing", NeurIPS, 2024.

Questions for authors

Please address the concerns raised in Weaknesses.

Reviewer_YMKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PATCH, a two-stage quantization error compensation framework for improving 2-bit LLM inference. Stage 1 applies fine-grained group-wise scaling and bias correction to capture heterogeneous quantization errors within individual output channels. Stage 2 freezes these corrections and jointly optimizes one regulation coefficient for each linear layer using an end-to-end KL objective, accounting for error propagation across layers. The resulting coefficients are fused into existing quantization scales, avoiding an additional correction branch during inference. Experiments cover Llama-3, Qwen3, and Gemma-2 models with GPTQ, AWQ, and QuIP. Results show consistent improvements in WikiText-2 perplexity and zero-shot accuracy over the evaluated compensation methods, while keeping optimization time and deployment overhead relatively low. The paper also includes ablations on compensation granularity, regulation granularity, optimization budget, and hardware latency across devices.

Critical strengths
  1. The paper addresses an important practical problem in extreme low-bit LLM quantization, where severe accuracy degradation still limits efficient deployment on resource-constrained hardware.
  2. The two-stage design is reasonably motivated, separating fine-grained local compensation from end-to-end strength regulation to account for quantization-error propagation across different transformer layers.
  3. The experimental evaluation is fairly broad, covering multiple model families, PTQ backbones, several ablations, optimization costs, metadata overhead, and GPU and NPU latency.
Critical weaknesses
  1. Group-wise compensation provides only modest additional accuracy once the same Stage 2 regulation is applied.
  2. Stage 1 sometimes worsens perplexity compared with the original quantized baseline, making Stage 2 appear responsible for a large part of the final improvement.
  3. The gap to BF16 remains quite large on several models, making the high-fidelity positioning somewhat stronger than the current results support.
  4. The deployment evaluation mainly reports normalized latency on two models and two hardware platforms, while absolute latency, detailed runtime settings, and the actual representation for PTQ methods such as QuIP are not sufficiently explained.
Questions for authors

Overall, I think this paper addresses an important problem in extreme low-bit LLM quantization, and the proposed idea is interesting. I have a few comments and concerns as follows. The group-wise compensation is reasonably motivated because one channel-wise coefficient cannot capture different errors across groups in the same output channel. The second stage is also useful, since optimizing layer-wise compensation strengths with an end-to-end objective considers interactions between quantized layers. The experiments cover multiple model families and PTQ methods, which provides a useful evaluation of generalization. However, several 2-bit results remain far from BF16 performance, so the high-fidelity positioning seems somewhat stronger than what the reported accuracy supports. Table 20 shows that after applying the same Stage 2 regulation, group-wise compensation improves average zero-shot accuracy over channel-wise compensation by only 0.28, 0.55, and 0.87 percentage points under GPTQ, AWQ, and QuIP. This makes it difficult to determine whether fine-grained correction itself is the main source of the overall improvement.

Can the authors provide further analysis of the independent contribution of group-wise compensation in PATCH? The relatively small margin over channel-wise compensation after Stage 2 makes the added group-wise granularity less compelling. Another concern is that Stage 1 alone sometimes produces worse perplexity than the original quantized baseline. Under GPTQ, for example, Llama-3.1-8B increases from 292.1 to 347.6 and Qwen3-8B from 59.34 to 87.64, before Stage 2 substantially improves both models. This supports the motivation about local-to-global mismatch, but it also suggests that end-to-end regulation may carry much of the performance gain attributed to the overall framework in the current experiments. How does PATCH compare against directly optimizing the quantization scales end-to-end under the same budget? A direct ablation on preserving the Stage 1 compensation direction would further strengthen this point. Can the authors report absolute prefill and decode latency and clarify metadata requirements across PTQ methods and hardware platforms?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.