acceptodds
Under review as a conference paper at ICLR 2027

GPTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Abstract

Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents GPTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, GPTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that GPTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Quantization, Model Compression, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WfFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

G² PTQ extends GPTQ-style weight compensation by incorporating both first- and second-order information under block-wise optimization objectives, using MSE for intermediate blocks and KL divergence for the final block. It refreshes gradient and Hessian estimates before quantizing each block, introduces trust-region scaling to stabilize gradient compensation, and derives efficient implementations. Experiments demonstrate improved perplexity and downstream accuracy over existing PTQ baselines.

Critical strengths
  1. The paper integrates first-order gradient compensation with second-order information under block-wise supervision, incorporating refreshed gradient and Hessian estimates and trust-region scaling. The efficient implementation makes this more comprehensive optimization framework practical for large-scale PTQ.
  2. The paper evaluates diverse LLM architectures, including large-scale MoE models, demonstrating improvements in downstream accuracy and distribution alignment. Detailed ablation studies validate the contributions of individual components, while runtime and memory measurements provide useful evidence of practical feasibility.
Critical weaknesses
  1. The key components of G² PTQ largely build on existing techniques. First-order gradient compensation was introduced by FOEM (AAAI 2026), and block-wise optimization objectives have been extensively studied in prior work such as BRECQ (ICLR 2021) and aespa (Towards next-level post-training quantization of hyper-scale transformers, NeurIPS 2024). The ablation study shows that most of the improvement over GPTQ comes from block-wise alignment, while the additional gains from other components are relatively modest. Moreover, the lack of a direct comparison with FOEM makes it difficult to assess the incremental contribution of the proposed method.
  2. G$^2$PTQ requires substantially more calibration memory and computation due to additional backward passes and Hessian estimation. For an 8B model, calibration memory increases from 4.62 GB to 14.57 GB, while runtime increases from 0.43 to 1.49 hours. These overheads are likely to become more pronounced for larger models. Without more convincing evidence of improvements under matched computational budgets, it remains unclear whether the additional complexity is justified by the performance gains.
Questions for authors
  1. Can the authors rewrite the key contribution of G$^2$PTQ over prior work? Which components represent genuinely new contributions rather than combinations or extensions of existing techniques?
  2. Can the authors provide a direct comparison with FOEM under matched experimental settings? In particular, how much improvement comes from the proposed components beyond the already established block-wise optimization objective?
  3. Theorem 3 derives a per-coordinate scaling factor, whereas Equation (56) averages these factors within each row. Does the row-wise averaged factor still satisfy the loss-change bound in Equation (50)?
9 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Q1. (Reviewer's W1 and Q1) Can the authors rewrite the key contribution of GPTQ over prior work? Which components represent genuinely new contributions rather than combinations or extensions of existing techniques?

A1. G²PTQ is not merely a combination of existing techniques. To the best of our knowledge, it is the first unified PTQ framework that integrates refreshed exact first- and second-order compensation under a global block-wise objective.

Specifically, our novel technical contributions include:

  1. Refreshed information with a block-wise objective. We solve the stale Hessian problem inherent in prior global-objective methods (e.g. OAC and GuidedQuant) without incurring prohibitive costs. Within this framework, we derive an efficient Hessian approximation for block-wise MSE/KL that estimates the Hessian in a single backpropagation pass (Section 4.1, lines 213-243).
  2. Exact gradient compensation and Trust-region scaling. We avoid the unnecessary approximation error in FOEM (Section 4.1, lines 245-252). Furthermore, we identify the exploding gradient step issue that uniquely arises when combining exact gradients with a global objective (Section 4.1, lines 251-255), and derive a closed-form trust-region scaling factor to guarantee stability (Section 4.1, lines 255-305).
  3. Efficient implementation. We provide a recursive implementation (Algorithm 1) that fully integrates G²PTQ into the GPTQ framework with minimal modifications while strictly preserving the asymptotic complexity of vanilla GPTQ.

Section 2 discusses GPTQ variants in detail. Below, we further summarize the key differences between G²PTQ and the existing work mentioned by the reviewer:

  • Objective approximation. Although prior work such as BRECQ and aespa also explores block-wise optimization, these methods optimize on the second-order approximation rather than the more precise first- and second-order approximation used in G²PTQ. We also derive an efficient approximation of the block-wise MSE Hessian in Theorem 1 without resorting to the coarser diagonal approximation used in BRECQ.
  • Optimization method. Whereas BRECQ and aespa use gradient descent over the full weight matrix for adaptive weight rounding (following AdaRound), G²PTQ optimizes the block-wise objective using closed-form, GPTQ-style weight compensation.

Q2. (Reviewer's W1 and Q2) Can the authors provide a direct comparison with FOEM under matched experimental settings? In particular, how much improvement comes from the proposed components beyond the already established block-wise optimization objective?

A2. Block-wise optimization is one of the core contributions of G²PTQ. As detailed in A1, G²PTQ centers on block-wise optimization within the GPTQ framework, using refreshed, exact first- and second-order compensation. To the best of our knowledge, block-wise optimization within the GPTQ framework has not previously been explored.

The other components also matter. We provide additional ablations to complement Table 4. Although block-wise second-order compensation alone does not necessarily improve upon the layer-wise GPTQ baseline because of stale information, incorporating refreshed information consistently improves quantization accuracy. Moreover, as noted in Section 5.3, lines 455–457, “this refreshing mechanism serves as a strict prerequisite for gradient compensation.” Exact gradient compensation, together with KL alignment in the last block, yields further significant improvements—for example, a 6.42% increase in QA accuracy for 2-bit quantization of Qwen3-8B.

(Qwen3-0.6B, W3A16)

MethodKL ↓PPL ↓QA ↑
GPTQ1.50e+0055.6033.09
+Block-wise Align5.07e-0118.7337.40
+Refreshed Info4.51e-0117.2738.61
+Exact Grad3.63e-0117.4839.15
+KL3.33e-0116.1239.83

(Qwen3-8B, W2A16)

MethodKL ↓PPL ↓QA ↑
GPTQ7.55e-0111.4548.60
+Block-wise Align1.68e+0150320998.6729.15
+Refreshed Info6.86e-0110.5649.50
+Exact Grad6.06e-0111.7954.85
+KL5.26e-019.8355.92
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Q2. (Reviewer's W2 and Q1) The additional calibration cost lacks sufficient generalization evidence... Generative evaluations cover only one 125B MoE model at 4 bits, leaving capability preservation under the main low-bit dense-model settings unclear.

A2 (continued from Part 1).

Comparison with FOEM.

  • Ablation of exact gradient compensation (Appendix E.1, lines 1324–1335). As shown in Table 8, replacing G²PTQ's exact gradient compensation with FOEM's approximate gradient compensation noticeably degrades both distribution alignment and downstream task performance, resulting in a 2.09% decrease in QA accuracy. These results underscore the importance of G²PTQ's exact gradient compensation.

  • Direct comparison with FOEM. We previously reproduced FOEM following the official code, but found that it failed to outperform GPTQ. This finding has also been reported in recent work [1] reproducing FOEM. We therefore did not include it as a baseline and instead carefully selected other strong baselines, such as GPTAQ and GuidedQuant. As shown below, G²PTQ significantly outperforms FOEM across different models and bit widths.

ModelMethodW2A16 KL ↓W2A16 PPL ↓W2A16 QA ↑W3A16 KL ↓W3A16 PPL ↓W3A16 QA ↑W4A16 KL ↓W4A16 PPL ↓W4A16 QA ↑
Qwen3 0.6BGPTQ4.73e+001.58e+0329.921.50e+0055.6033.094.58e-0118.3341.57
FOEM9.52e+001.55e+0529.111.89e+00128.7533.275.05e-0132.8944.19
G²PTQ1.55e+0064.9931.723.87e-0116.9239.221.17e-0112.6345.85
G²PTQ*1.26e+0053.2532.933.33e-0116.1239.831.05e-0112.5245.80
Qwen3.5 2BGPTQ1.42e+0039.9532.182.28e-018.7051.296.36e-027.0056.57
FOEM2.99e+00248.7731.073.87e-0118.6151.069.53e-0213.4357.65
G²PTQ1.05e+0025.0636.091.99e-018.4251.875.83e-026.9857.22
G²PTQ*8.42e-0119.9937.751.77e-018.2053.535.37e-026.9857.97
LLaMA3 8BGPTQ1.15e+0018.1138.611.69e-014.9964.634.99e-024.3368.44
FOEM3.62e+00235.1832.223.32e-018.6162.389.30e-026.7368.03
G²PTQ8.65e-0114.7647.471.29e-014.7866.073.94e-024.2869.06
G²PTQ*6.12e-019.1851.161.22e-014.7566.563.75e-024.2769.41

[1] Lee, Gunjun, et al. "SchurQuant: Groupwise Discrete Optimization for Layer-Wise LLM Quantization." arXiv preprint arXiv:2608.15567 (2026).

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3)

Q3. (Reviewer's W2) GPTQ requires substantially more calibration memory and computation due to additional backward passes and Hessian estimation... These overheads are likely to become more pronounced for larger models.

A3. The quantization overhead is an acceptable, one-time cost. Once the model is quantized, the higher accuracy benefits all downstream applications, making the upfront cost well worth it.

Although G²PTQ incurs additional calibration costs compared with GPTQ, a highly efficient algorithm for LLM quantization, the overhead remains within a reasonable budget. As shown in Table 5, G²PTQ can quantize an 8B model in 1.49 hours on a single Hopper GPU. The G²PTQ* variant takes an additional 0.75 hours.

Moreover, G²PTQ can scale to 100B+ models. During the rebuttal period, we substantially improved the efficiency of G²PTQ by optimizing multi-GPU communication and computation workflows, building on the efficiency optimizations described in Appendix C.3. The table below reports the quantization overhead of the optimized G²PTQ* implementation, supplementing Appendix E.2, lines 1338–1346.

  1. Memory cost. The maximum GPU memory required by G²PTQ is bounded by backpropagating a single Transformer block, not the entire model. As shown below, even a massive 753B model can be quantized on 140GB GPUs.
  2. Time cost. The time required remains well within practical limits for an offline procedure. Specifically, a 753B model can be quantized in 6 hours using only 32 Hopper GPUs, or in less than a day using only 8 Hopper GPUs.
  3. Calibration dataset size. As discussed in Appendix E.1, lines 1290–1294, G²PTQ is highly sample-efficient and requires as few as 128 calibration samples to outperform GPTQ.

(GLM-5.2, 753B)

#GPUsMemory (GPTQ / G²PTQ*)Time (GPTQ / G²PTQ*)
842.03 / 97.38 GiB9.69 / 17.51 hours
3242.03 / 69.46 GiB4.70 / 6.00 hours

(As noted in Appendix E.2, lines 1338–1346, we quantize MoE models using 8K calibration samples with 2K length. The table reports per-GPU memory usage and time. For the 753B model, the overhead is estimated from quantizing the first 10 of its 78 Transformer blocks.)

Reviewer_6e2fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

G²PTQ extends GPTQ-style post-training quantization with block-level reconstruction objectives, refreshed gradient and approximate Hessian estimates, and scaled first-order compensation. It also derives efficient recursive weight updates. Experiments across dense and MoE models evaluate distributional alignment, perplexity, downstream accuracy, component contributions, and calibration costs.

Critical strengths

The experiments cover diverse model families, including dense and MoE architectures, across multiple quantization settings. The reported improvements over established baselines, particularly at low bit-widths, provide empirical evidence that the approach can improve quantization accuracy beyond a single model or configuration.

Critical weaknesses
  1. The methodological contribution is insufficiently isolated. Asymmetric calibration and scaled first-order compensation have precedents in GPTAQ and FOEM. Under GPTQ’s ideal quadratic objective, exact compensation leaves zero gradient on the remaining free weights. The paper should clarify why its nonlinear block objective requires additional compensation and better isolate its benefits from block-level alignment and information refreshing.
  2. The additional calibration cost lacks sufficient generalization evidence. Backpropagation and per-block hyperparameter search substantially increase calibration costs, but robustness to calibration data and tuning choices is insufficiently examined. Generative evaluations cover only one 125B MoE model at 4 bits, leaving capability preservation under the main low-bit dense-model settings unclear.
Questions for authors

Given the substantial additional calibration cost from backpropagation and hyperparameter search, what evidence demonstrates a favorable accuracy–cost trade-off over existing methods? Does the advantage persist under comparable calibration budgets and on held-out generative benchmarks?

10 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Q1. (Reviewer's W1) The methodological contribution is insufficiently isolated... The paper should clarify why its nonlinear block objective requires additional compensation and better isolate its benefits from block-level alignment and information refreshing.

A1.

  • First-order compensation matters. As detailed in Section 4.1, we optimize a block-wise objective (MSE/KL) that measures distributional alignment. When Transformer blocks are quantized sequentially, distributional distortion leads to a nonzero gradient term that cannot be ignored.
  • Key difference in first-order compensation. While FOEM uses approximate first-order compensation, we prove in Appendix B.2 that this approximation introduces unnecessary error. We therefore use exact gradient compensation, coupled with trust-region scaling to prevent the exploding updates specific to exact gradient compensation (Section 4.1, lines 245–305). The empirical results in Table 8 show that exact compensation is critical, yielding a 2.09% increase in QA accuracy.
  • Detailed ablations. Table 4 isolates each component of G²PTQ and demonstrates its effectiveness.

(Qwen3-0.6B, W3A16, Table 4)

MethodKL ↓PPL ↓QA ↑
GPTQ1.50e+0055.6033.09
+Block-wise Align5.07e-0118.7337.40
+Refreshed Info4.51e-0117.2738.61
+Exact Grad3.63e-0117.4839.15
+KL3.33e-0116.1239.83

Q2. (Reviewer's W2 and Q1) The additional calibration cost lacks sufficient generalization evidence... Generative evaluations cover only one 125B MoE model at 4 bits, leaving capability preservation under the main low-bit dense-model settings unclear.

A2. G²PTQ generalizes well across multiple bit widths and models.

  • Extensive evaluation. As the reviewer also notes in S1, "the experiments cover diverse model families, including dense and MoE architectures, across multiple quantization settings" (Tables 12–17). We also evaluate a broad range of metrics, including distributional alignment (KL) and downstream performance (PPL, QA), to avoid biased evaluation.
  • Reasoning tasks. To directly address the reviewer's concern, we additionally evaluate a low-bit dense model on reasoning tasks, complementing Table 18. G²PTQ significantly outperforms the baselines, demonstrating its generalization capability.

(Qwen3-8B W2A16)

MethodGSM8KMATH-500IFBench
GPTQ37.30±1.6624.87±1.558.33±0.34
GuidedQuant27.09±0.9515.80±1.0610.11±0.70
GPTAQ55.92±1.1434.80±0.6012.68±1.23
G²PTQ*71.29±1.1456.40±2.8215.58±1.73
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Q3. (Reviewer's W2 and Q1) Given the substantial additional calibration cost from backpropagation and hyperparameter search, what evidence demonstrates a favorable accuracy–cost trade-off over existing methods?

A3. The quantization overhead is an acceptable, one-time cost. Once the model is quantized, the higher accuracy benefits all downstream applications, making the upfront cost well worth it.

Although G²PTQ incurs additional calibration costs compared with GPTQ, a highly efficient algorithm for LLM quantization, the overhead remains within a reasonable budget. As shown in Table 5, G²PTQ can quantize an 8B model in 1.49 hours on a single Hopper GPU. The G²PTQ* variant takes an additional 0.75 hours.

Moreover, G²PTQ can scale to 100B+ models. During the rebuttal period, we substantially improved the efficiency of G²PTQ by optimizing multi-GPU communication and computation workflows, building on the efficiency optimizations described in Appendix C.3. The table below reports the quantization overhead of the optimized G²PTQ* implementation, supplementing Appendix E.2, lines 1338–1346.

  1. Memory cost. The maximum GPU memory required by G²PTQ is bounded by backpropagating a single Transformer block, not the entire model. As shown below, even a massive 753B model can be quantized on 140GB GPUs.
  2. Time cost. The time required remains well within practical limits for an offline procedure. Specifically, a 753B model can be quantized in 6 hours using only 32 Hopper GPUs, or in less than a day using only 8 Hopper GPUs.
  3. Calibration dataset size. As discussed in Appendix E.1, lines 1290–1294, G²PTQ is highly sample-efficient and requires as few as 128 calibration samples to outperform GPTQ.

(GLM-5.2, 753B)

#GPUsMemory (GPTQ / G²PTQ*)Time (GPTQ / G²PTQ*)
842.03 / 97.38 GiB9.69 / 17.51 hours
3242.03 / 69.46 GiB4.70 / 6.00 hours

(As noted in Appendix E.2, lines 1338–1346, we quantize MoE models using 8K calibration samples with 2K length. The table reports per-GPU memory usage and time. For the 753B model, the overhead is estimated from quantizing the first 10 of its 78 Transformer blocks.)


Q4. (Reviewer's W2 and Q1) Does the advantage persist under comparable calibration budgets...

A4. We further compare G²PTQ with another strong baseline, YAQA [1], to show that G²PTQ can outperform prior work with a significantly smaller computational budget.Specifically, as described in Section 2, lines 112–113, YAQA extends GPTQ's layer-wise loss to a global objective, similar to GuidedQuant. Whereas G²PTQ uses only 1K samples with a sequence length of 2K, we follow the original YAQA paper and use 64K samples for YAQA to achieve its best accuracy.

  • As shown in the table, YAQA takes 6.75 hours on a single H20 GPU to quantize Qwen3-0.6B, whereas G²PTQ achieves similar accuracy in just 0.18 hours. The G²PTQ* variant consistently outperforms YAQA while requiring only 4.89% of its calibration time.

(W3A16)

ModelMethodKL ↓PPL ↓QA ↑
Qwen3-0.6BRTN7.44e+0016173.3329.68
GPTQ1.50e+0055.6033.09
GuidedQ4.96e-0117.7638.28
GPTAQ8.92e-0130.4034.58
YAQA3.74e-0116.9139.21
G²PTQ3.87e-0116.9239.22
G²PTQ*3.33e-0116.1239.83
Qwen3.5-0.8BRTN1.89e+0080.8932.71
GPTQ3.02e-0112.3442.81
GuidedQ3.12e-0112.7843.43
GPTAQ2.78e-0112.4142.96
YAQA2.55e-0112.3043.00
G²PTQ2.58e-0111.7542.78
G²PTQ*2.21e-0111.3244.47

(We reproduce YAQA under the same settings as G²PTQ, using the same scalar quantizer for a fair comparison. We use the Sketch-B variant for its superior accuracy. Because YAQA's original two-sided compensation is extremely slow due to its quartic complexity, we implement GPTQ-2D [2] for weight compensation.)

[1] Tseng, Albert, et al. “Model-Preserving Adaptive Rounding.” Forty-third International Conference on Machine Learning, 2026, https://openreview.net/forum?id=PKFilPWjMI.

[2] Chen, Jiale, Torsten Hoefler, and Dan Alistarh. "GPTQ-2D: Cubic-time two-sided adaptive rounding." arXiv preprint arXiv:2607.27042 (2026).

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3)

Q5. (Reviewer's W2) Robustness to calibration data and tuning choices is insufficiently examined.

A5. G²PTQ is robust to calibration data and tuning choices.

  • Robustness to calibration data. An ablation study on calibration set size is provided in Appendix E.1, lines 1290–1294. While G²PTQ generally requires more calibration samples than GPTQ to achieve its best accuracy, it still outperforms GPTQ with as few as 128 samples. We also conduct an ablation study on the calibration domain. The following table demonstrates G²PTQ's robustness across calibration domains.

(Qwen3-0.6B)

Calibration SetMethodW2A16 KL ↓W2A16 PPL ↓W2A16 QA ↑W3A16 KL ↓W3A16 PPL ↓W3A16 QA ↑W4A16 KL ↓W4A16 PPL ↓W4A16 QA ↑
NeuralMagicRTN1.43e+017.56e+0628.957.44e+001.62e+0429.685.84e-0117.6141.36
GPTQ4.73e+001.58e+0329.921.50e+0055.6033.094.58e-0118.3341.57
GuidedQuant2.46e+00143.6730.054.96e-0117.7638.281.42e-0112.8845.41
GPTAQ3.33e+00364.9628.618.92e-0130.4034.583.03e-0115.8542.85
G²PTQ1.55e+0064.9931.723.87e-0116.9239.221.17e-0112.6345.85
G²PTQ*1.26e+0053.2532.933.33e-0116.1239.831.05e-0112.5245.80
Open PerfectblendRTN1.43e+017.56e+0628.977.44e+001.62e+0429.655.84e-0117.6141.36
GPTQ5.10e+002.13e+0328.841.51e+0054.2032.725.16e-0119.3842.13
GuidedQuant3.14e+00205.1930.374.86e-0117.6839.571.30e-0112.5645.89
GPTAQ2.93e+00235.2431.159.89e-0133.9034.253.30e-0116.1142.23
G²PTQ2.82e+00162.2232.763.71e-0116.4540.261.08e-0112.4046.15
G²PTQ*1.72e+0066.7433.913.09e-0115.6341.129.47e-0212.2747.12
Fineweb EduRTN1.43e+017.56e+0628.977.44e+001.62e+0429.655.84e-0117.6141.36
GPTQ6.05e+005.21e+0330.101.50e+0047.5133.364.41e-0117.3542.36
GuidedQuant3.34e+00195.2531.895.89e-0116.0239.901.98e-0112.4946.23
GPTAQ3.95e+00600.1430.629.32e-0125.3636.072.95e-0114.4842.82
G²PTQ2.57e+00115.6133.975.10e-0115.4340.511.71e-0112.1147.57
G²PTQ*1.83e+0048.0934.624.43e-0114.5842.181.55e-0112.2446.84
  • Robustness to tuning choices. As detailed in Appendix C.2, we use the same auto-tuning strategy throughout, without tuning it specifically for individual settings. Experiments across different bit widths and models demonstrate its robustness.
Reviewer_2UpnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel PTQ framework that combines first-order gradient compensation with approximate second-order curvature using block-wise MSE objectives and a final-block KL divergence objective. The method refreshes gradient and curvature estimates during quantization, introduces adaptive scaling to stabilize gradient compensation, and derives efficient recursive and batched weight updates. Experiments across dense and MoE language models show improved distributional alignment and downstream accuracy over the evaluated baselines. But its calibration cost has increased compared to the baseline.

Critical strengths
  1. The proposed method is well-motivated. It is an improvement upon the widely used GPTQ and holds clear practical value. It is sound to perform block-wise optimization.
  2. The experiments are quite comprehensive. It covers both dense models and MoE models, and also covers a range of model sizes and bit widths.
Critical weaknesses
  1. Some baselines most relevant to the core contribution are missing. For example, the paper takes the first-order compensation of FOEM as its primary starting point, yet fails to provide a direct comparison with the original FOEM.
  2. The applicability of the G²PTQ* variant remains unclear. In some cases, G²PTQ* performs worse than G²PTQ, despite requiring more frequent gradient and curvature estimation. The paper does not explain these results or provide guidance on choosing between the two variants.
  3. The actual resource overhead for calibration is significant. In Table 5, the authors only evaluate the calibration overhead for small-scale models (0.6B and 8B). Although they also report the calibration overhead for Qwen3.8-Flash-Next (125B) in the appendix (E.2), this cost is so high that it may be difficult to scale up to larger models.
Questions for authors
  1. Could the authors provide comparisons between G²PTQ and FOEM under matched experimental settings?
  2. What explains the performance regressions of G²PTQ* relative to G²PTQ, and under what conditions do its benefits justify the additional calibration cost?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1)

Q1. (Reviewer's W1 and Q1) Some baselines most relevant to the core contribution are missing... fails to provide a direct comparison with the original FOEM.

A1. Thanks for the suggestion. In fact, we previously reproduced FOEM following the official code, but found that it failed to outperform GPTQ. This finding has also been reported in recent work [1] reproducing FOEM. We therefore did not include it as a baseline and instead carefully selected other strong baselines, such as GPTAQ and GuidedQuant.

As shown below, G²PTQ significantly outperforms FOEM across different models and bit widths.

ModelMethodW2A16 KL ↓W2A16 PPL ↓W2A16 QA ↑W3A16 KL ↓W3A16 PPL ↓W3A16 QA ↑W4A16 KL ↓W4A16 PPL ↓W4A16 QA ↑
Qwen3 0.6BGPTQ4.73e+001.58e+0329.921.50e+0055.6033.094.58e-0118.3341.57
FOEM9.52e+001.55e+0529.111.89e+00128.7533.275.05e-0132.8944.19
G²PTQ1.55e+0064.9931.723.87e-0116.9239.221.17e-0112.6345.85
G²PTQ*1.26e+0053.2532.933.33e-0116.1239.831.05e-0112.5245.80
Qwen3.5 2BGPTQ1.42e+0039.9532.182.28e-018.7051.296.36e-027.0056.57
FOEM2.99e+00248.7731.073.87e-0118.6151.069.53e-0213.4357.65
G²PTQ1.05e+0025.0636.091.99e-018.4251.875.83e-026.9857.22
G²PTQ*8.42e-0119.9937.751.77e-018.2053.535.37e-026.9857.97
LLaMA3 8BGPTQ1.15e+0018.1138.611.69e-014.9964.634.99e-024.3368.44
FOEM3.62e+00235.1832.223.32e-018.6162.389.30e-026.7368.03
G²PTQ8.65e-0114.7647.471.29e-014.7866.073.94e-024.2869.06
G²PTQ*6.12e-019.1851.161.22e-014.7566.563.75e-024.2769.41

[1] Lee, Gunjun, et al. "SchurQuant: Groupwise Discrete Optimization for Layer-Wise LLM Quantization." arXiv preprint arXiv:2608.15567 (2026).


Q2. (Reviewer's W2 and Q2) The applicability of the G²PTQ* variant remains unclear... The paper does not explain these results or provide guidance on choosing between the two variants.

A2. Thanks for the constructive feedback. We recommend G²PTQ* when preserving accuracy is the priority, particularly for low-bit quantization. G²PTQ offers a less expensive alternative.

  1. G²PTQ* generally outperforms G²PTQ with more accurate first- and second-order information. This aligns with our observation in lines 204–211 that "the gradient and Hessian should reflect the current state of the model." Specifically, as shown in Tables 12–14, G²PTQ* outperforms G²PTQ in average KL for 13/13, 12/13, and 11/13 models, and in QA accuracy for 11/13, 9/13, and 10/13 models, at 2, 3, and 4 bits, respectively.
  2. The additional calibration cost is acceptable in practice. While G²PTQ*'s calibration time is approximately 1.67 times that of G²PTQ (Table 5), this overhead is an acceptable, one-time cost (see A3 below). The additional calibration cost is incurred only once, while the accuracy gains carry over to downstream applications.
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2)

Q3. (Reviewer's W3) The actual resource overhead for calibration is significant... it may be difficult to scale up to larger models.

A3. The quantization overhead is an acceptable, one-time cost. Once the model is quantized, the higher accuracy benefits all downstream applications, making the upfront cost well worth it.

Although G²PTQ incurs additional calibration costs compared with GPTQ, a highly efficient algorithm for LLM quantization, the overhead remains within a reasonable budget. As shown in Table 5, G²PTQ can quantize an 8B model in 1.49 hours on a single Hopper GPU. The G²PTQ* variant takes an additional 0.75 hours.

Moreover, G²PTQ can scale to 100B+ models. During the rebuttal period, we substantially improved the efficiency of G²PTQ by optimizing multi-GPU communication and computation workflows, building on the efficiency optimizations described in Appendix C.3. The table below reports the quantization overhead of the optimized G²PTQ* implementation, supplementing Appendix E.2, lines 1338–1346.

  1. Memory cost. The maximum GPU memory required by G²PTQ is bounded by backpropagating a single Transformer block, not the entire model. As shown below, even a massive 753B model can be quantized on 140GB GPUs.
  2. Time cost. The time required remains well within practical limits for an offline procedure. Specifically, a 753B model can be quantized in 6 hours using only 32 Hopper GPUs, or in less than a day using only 8 Hopper GPUs.
  3. Calibration dataset size. As discussed in Appendix E.1, lines 1290–1294, G²PTQ is highly sample-efficient and requires as few as 128 calibration samples to outperform GPTQ.

(GLM-5.2, 753B)

#GPUsMemory (GPTQ / G²PTQ*)Time (GPTQ / G²PTQ*)
842.03 / 97.38 GiB9.69 / 17.51 hours
3242.03 / 69.46 GiB4.70 / 6.00 hours

(As noted in Appendix E.2, lines 1338–1346, we quantize MoE models using 8K calibration samples with 2K length. The table reports per-GPU memory usage and time. For the 753B model, the overhead is estimated from quantizing the first 10 of its 78 Transformer blocks.)

AuthorsOpenReview ↗

Gentle Reminder — response to your review

Dear Reviewer 2Upn,

Thank you for the detailed review.

With two days remaining in the discussion period, we would like to follow up and ask whether our responses have adequately addressed your questions and concerns. If any points remain unresolved or require further clarification, we would welcome the opportunity to address them.

Thank you again for your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

We appreciate the reviewers’ thorough assessment and helpful suggestions. The key highlights are outlined below, with detailed results and supporting references provided in our responses to each reviewer. To facilitate further review, we have also made our code publicly available at https://anonymous.4open.science/r/G2PTQ-5453/.

  • Key contributions (Reviewer 6e2f’s A1, A6–8; Reviewer WfFj’s A1–2, A6, A8). To the best of our knowledge, G²PTQ is the first unified PTQ framework to integrate refreshed, exact first- and second-order compensation under a global block-wise objective. Specifically, our novel technical contributions include:

    1. Refreshed information with a block-wise objective. We address the stale Hessian problem inherent in prior global-objective methods (e.g., OAC and GuidedQuant) without incurring prohibitive costs. Within this framework, we derive an efficient Hessian approximation for block-wise MSE/KL objectives that estimates the Hessian in a single backpropagation pass (Section 4.1, lines 213–243).

    2. Exact gradient compensation and trust-region scaling. We avoid the unnecessary approximation error introduced by FOEM (Section 4.1, lines 245–252). Furthermore, we identify the issue of exploding gradient steps that arises specifically when combining exact gradients with a global objective (Section 4.1, lines 251–255) and derive a closed-form trust-region scaling factor to guarantee stability (Section 4.1, lines 255–305). We provide additional ablations on four models across three bit widths to demonstrate the effectiveness of the proposed exact gradient compensation (see A8 in our response to Reviewer 6e2f).

    3. Efficient implementation. We provide a recursive implementation (Algorithm 1) that fully integrates G²PTQ into the GPTQ framework with minimal modifications while strictly preserving the asymptotic complexity of vanilla GPTQ.

  • Calibration overhead (Reviewer 2Upn’s A3; Reviewer 6e2f’s A3, A9; Reviewer WfFj’s A3, A8). The quantization overhead is an acceptable, one-time cost. Once the model is quantized, its higher accuracy benefits all downstream applications, making the upfront cost worthwhile. Although G²PTQ incurs additional calibration costs compared with GPTQ, a highly efficient algorithm for LLM quantization, the overhead remains reasonable. G²PTQ can quantize an 8B model in 1.29 hours on a single Hopper GPU (see the table in our response to Reviewer WfFj’s A8), comparable to GPTAQ (1.25 hours). The G²PTQ* variant takes an additional 0.56 hours. Moreover, G²PTQ scales to models with over 100B parameters. A 753B model can be quantized in 6 hours using only 32 Hopper GPUs, comparable to GPTQ (4.7 hours).

  • Additional experiments

    • (Reviewer 2Upn’s A1; Reviewer WfFj’s A2) Comparison with FOEM. We reproduced FOEM using the official code. The results show that G²PTQ significantly outperforms FOEM across different models and bit widths.

    • (Reviewer 6e2f’s A4; Reviewer WfFj’s A4) Comparison with YAQA. We reproduced YAQA Sketch-B using GPTQ-2D for two-sided quantization. The G²PTQ* variant consistently outperforms YAQA while requiring only 4.89% of its calibration time.

    • (Reviewer 6e2f’s A2) Additional generative evaluations. We conducted additional experiments on 2-bit quantization of Qwen3-8B, in which G²PTQ significantly outperforms the baselines (e.g., 13.29% higher average accuracy than GPTAQ).

    • (Reviewer 6e2f’s A5) Robustness to calibration data. We conducted an ablation study across calibration domains, demonstrating G²PTQ’s robustness to the choice of calibration domain.

  • Clarifications

    • (Reviewer 2Upn’s A2) Guidance on choosing between the G²PTQ and G²PTQ* variants. We recommend G²PTQ* when preserving accuracy is the priority, particularly for low-bit quantization. G²PTQ offers a less expensive alternative.

    • (Reviewer 6e2f’s A5) Robustness to tuning choices. We use the same auto-tuning strategy throughout, without tailoring it to individual settings.

    • (Reviewer WfFj’s A5) Theoretical guarantee for the row-wise averaged factor. Although the per-coordinate scaling factor is theoretically sound, our early experiments showed that averaging yields similar performance while allowing a simpler implementation. We therefore use the row-wise averaged factor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.