acceptodds
Under review as a conference paper at ICLR 2027

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

Abstract

Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade model performance on the same calibration data. Our analysis further shows that weights with lower reconstruction loss on calibration data can have higher loss than other weights when the distribution of input activations changes. Motivated by these observations and our analysis, we propose Distributionally Robust Quantization (DRQ), a post-hoc refinement process that minimizes worst-case reconstruction loss over a constrained set of input activation distributions. DRQ refines the integer codes representing quantized weights within the existing quantization grid, keeping quantization parameters and inference operators unchanged. Extensive experiments show that DRQ improves models quantized by six representative PTQ methods, including AWQ, GPTQ, and ParoQuant, and delivers gains across both dense and mixture-of-experts large language models. These results establish DRQ as a general post-hoc refinement framework for weight-only PTQ, achieving better downstream performance without adding inference overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training quantization, model compression, distributionally robust optimization, discrete optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RHohOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

"When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization" identifies the schematic problem of PTQ that minimization of the reconstruction loss per layer does not necssarliry means the minimization of the task loss. This paper mathematically shows when this discrepancy occurs and identifies that non-active directions in the calibration dataset matter. The authors also proposed a method to keep the quantization scale and zero point and change the integer code to achieve robustness to the distribution shift of input activation. The experiments shows in diferrennet PTQ methods and architectures the proposed method achieved better performance, and this method is a completely add-on post-hoc method to solve this discrepancy.

Critical strengths

This study is very impressive and very important for the PTQ community:

  1. Every researcher knows that reconstruction loss does not necessarily align with task loss. However, nobody deals with this problem as a central target of research. The authors ask it and provide a very large discrepancy between reconstruction loss and task loss.
  2. The proposed method is very easy to implement for existing PTQ.
  3. The authors conducted extensive experiments on several PTQ methods, and the experimental results are consistent.
Critical weaknesses

This paper is good, but readability is the problem.

  1. These authors clearly used an LLM heavily to write and organize this paper, as stated in the AI use statement. This is OK; however, some parts, especially the experimental settings in the appendix, are too dense for human readers. This is for LLM reading.
  2. The analysis of reconstruction loss and task loss is relatively weaker than the experiments of the proposed method.
  3. The proposed method is just a combination of existing methods, and the novely is the mainly focuses on the connection to PTQ.
Questions for authors

I hope the authors add a more comprehensive analysis of reconstruction loss and task loss. Different architectures have different calibration data, internal latent expression analysis, and activation analysis. Especially, I want to know whether the distribution shift really occurs.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RHoh (Part 1/3)

Thank you for recognizing our work. We also believe that this study is very helpful to the PTQ community. We hope the analyses and experiments below resolve your concerns and encourage you to raise your score.

1. Readability of the Appendix [Weakness 1]

We have reorganized and simplified Appendices C to E to make them concise and easy to scan for readers.

In the revised manuscript, we consolidated all experimental settings into a unified summary table (Table 4), extracted inline MoE statistics into Table 11, removed dense inline timing lists in Appendix E.2, and added a numbered four-step workflow with an illustrative one-round example in Appendix C.

2. Comprehensive Analysis of Reconstruction Loss and Task Loss [Weakness 2 & Question 1 (Part A)]

Lowering calibration reconstruction loss can hurt model performance through two distinct mechanisms: an objective mismatch between local residuals and global task gradients on identical inputs, and overfitting to the sampled search positions.

As shown in Figure 1 of the submitted paper (summarized in Table F1), reducing a single layer’s reconstruction loss frequently increases whole-model PPL, even on the exact same calibration data:

Table F1. Summary of Figure 1 single-layer modifications.

InitializationEvaluation dataUpdates with lower calibration Lrec\mathcal{L}_{\mathrm{rec}} but higher model PPL
AWQ W3Same WT2 calibration examples18 / 32 (56.3%)
AWQ W3Held-out WT2 training text413 / 560 (73.8%)
GPTQ W3Held-out WT2 training text264 / 560 (47.1%)

To see why this happens even without distribution shift, consider the same fixed evaluation examples before and after a single-layer weight update. Let r0r_0 denote the initial quantized layer’s output residual (W^0x−Wx\widehat W_0 x - Wx), and let δh\delta h be the layer-output change induced by the update. With input activations and all other layers fixed, the reconstruction-loss change and the end-to-end model-loss Taylor expansion are

\begin{aligned} \Delta L_{\mathrm{rec}} =\mathbb{E}\!\left[2\langle r_0,\delta h\rangle+\|\delta h\|^2\right], \qquad \Delta J =\mathbb{E}\!\left[\langle g_0,\delta h\rangle\right] +O\!\left(\mathbb{E}\|\delta h\|^2\right), \end{aligned}

where g0g_0 is the gradient of the model task loss with respect to the layer output at the initial quantized model.

These two equations evaluate the exact same output change δh\delta h against different vectors (r0r_0 versus g0g_0). Whenever an update δh\delta h satisfies 2E⟨r0,δh⟩<−E∣δh∣22\mathbb{E}\langle r_0,\delta h\rangle < -\mathbb{E}\\|\delta h\\|^2 while E⟨g0,δh⟩>0\mathbb{E}\langle g_0,\delta h\rangle > 0 dominates the higher-order remainder in ΔJ\Delta J, it strictly lowers layer-output reconstruction error while simultaneously worsening model NLL on the exact same inputs. Theorems 3.1, 3.2, and B.2 complement this analysis by showing how activation shifts reverse reconstruction rankings and why simply adding more calibration data cannot eliminate an existing population-level objective mismatch.

We verified both mechanisms on Llama-3.2-1B-Instruct with AWQ and GPTQ W3A16 (g=128g=128) across 560 single-layer updates (112 linear layers ×\times 5 updates per layer), evaluating both losses on the exact same text (Table R1).

Table R1. Same-text reconstruction and NLL mismatch across 560 single-layer updates.

MethodText used for both lossesUpdates lowering full-text Lrec\mathcal{L}_{\mathrm{rec}}Fraction increasing model NLL
AWQ W3WT2 calibration evaluation text560338 / 560 (60.36%)
AWQ W3WT2 test split560302 / 560 (53.93%)
AWQ W3C4558230 / 558 (41.22%)
GPTQ W3WT2 calibration evaluation text0Undefined
GPTQ W3WT2 test split31 / 3 (33.33%)
GPTQ W3C4287 / 28 (25.00%)

Takeaways from Table R1:

  • Same-text objective mismatch in AWQ: Over 60% of AWQ updates that reduce full-text calibration reconstruction loss simultaneously worsen model NLL on that exact same text, confirming the local-versus-global mismatch even without distribution shift.

  • Search-subset overfitting in GPTQ: All 560 GPTQ updates reduce reconstruction loss on the 512 sampled search positions, yet 100% of them increase reconstruction loss across the full calibration set, directly motivating DRQ’s checking-subset rules and directional penalty.

The candidate-selection comparison (Table R2) continues in Part 2.

AuthorsOpenReview ↗

Response to Reviewer RHoh (Part 2/3)

2. Comprehensive Analysis of Reconstruction Loss and Task Loss (continued) [Weakness 2 & Question 1 (Part A)]

This continues the Llama-3.2-1B-Instruct AWQ/GPTQ W3A16 (g=128g=128) single-layer analysis in Part 1.

Furthermore, when selecting from each layer’s candidate pool (Table R2), DRQ’s robust objective directly resolves this mismatch:

Table R2. Model NLL changes (nats/token) averaged over 112 single-layer interventions. Negative values mean lower (better) NLL. Each comparison reports its paired document-bootstrap 95% confidence interval.

Scenario / textRec −- InitialRobust −- Initial [95% CI]Robust −- Rec [95% CI]
GPTQ / Calibration WT2+0.00327334+0.00327334−0.00032223\mathbf{-0.00032223} [−0.00050077,−0.00012846-0.00050077,-0.00012846]−0.00359556\mathbf{-0.00359556} [−0.00416228,−0.00305981-0.00416228,-0.00305981]
GPTQ / WT2 test+0.00377090+0.00377090−0.00047485\mathbf{-0.00047485} [−0.00075216,−0.00021586-0.00075216,-0.00021586]−0.00424575\mathbf{-0.00424575} [−0.00501483,−0.00342377-0.00501483,-0.00342377]
GPTQ / C4+0.00532017+0.00532017−0.00036509\mathbf{-0.00036509} [−0.00058861,−0.00014868-0.00058861,-0.00014868]−0.00568526\mathbf{-0.00568526} [−0.00632973,−0.00502850-0.00632973,-0.00502850]
AWQ / Calibration WT2−0.00010412-0.00010412−0.00067508\mathbf{-0.00067508} [−0.00084005,−0.00050948-0.00084005,-0.00050948]−0.00057096\mathbf{-0.00057096} [−0.00069542,−0.00044752-0.00069542,-0.00044752]
AWQ / WT2 test+0.00007485+0.00007485−0.00073284\mathbf{-0.00073284} [−0.00096603,−0.00050755-0.00096603,-0.00050755]−0.00080769\mathbf{-0.00080769} [−0.00097886,−0.00062784-0.00097886,-0.00062784]
AWQ / C4+0.00072228+0.00072228−0.00052285\mathbf{-0.00052285} [−0.00073941,−0.00032451-0.00073941,-0.00032451]−0.00124513\mathbf{-0.00124513} [−0.00140055,−0.00107094-0.00140055,-0.00107094]

Takeaway from Table R2: Selecting candidates by minimizing reconstruction loss worsens model NLL in five of six cases, whereas DRQ’s robust objective significantly lowers model NLL relative to both the initial model and reconstruction selection across every calibration and held-out dataset.

3. Algorithmic Novelty and Broad Applicability [Weakness 3]

Seamless compatibility with existing PTQ methods is a major strength rather than a weakness: it gives DRQ the flexibility to serve as a general, future-proof refinement framework as PTQ continues to evolve. Rather than replacing specific quantizers, DRQ resolves a fundamental objective mismatch shared across weight-only PTQ pipelines. Because it operates directly on the quantized integer grid, DRQ remains broadly applicable as new PTQ algorithms are developed. Beyond this plug-in flexibility, our novelty lies in two main aspects:

  • Theoretical contribution: Figure 1 and Figure 4 show that lower reconstruction loss can degrade model PPL even as calibration data grow. Theorem 3.1 derives the exact condition under which activation second-moment shifts reverse reconstruction rankings, and Theorem 3.2 reduces the worst-case objective to a tractable spectral penalty.

  • Algorithmic and practical contribution: Existing PTQ methods differ widely in their weight transformations, scaling, clipping, and calibration workflows. DRQ provides a unified discrete search that updates only integer codes on the fixed quantization grid while preserving scales, zero points, and inference kernels. Across our paper and new rebuttal experiments, DRQ improves eight diverse PTQ methods (GPTQ, AWQ, RTN, OmniQuant, GPTAQ, ParoQuant, TesseraQ, and SignRoundV2) across dense and MoE models with zero inference overhead.

AuthorsOpenReview ↗

Response to Reviewer RHoh (Part 3/3)

4. Does the Distribution Shift Really Occur in Practice? [Question 1 (Part B)]

Yes. Direct activation measurements across datasets show that shifting from WikiText-2 to C4 or Pile strongly amplifies activation energy along DRQ’s worst-case direction and sharply increases local reconstruction loss, whereas within-domain controls remain near zero.

Tables R3 and R4 measure activation shifts and their impact on layer reconstruction loss with fixed quantized weights on Llama-3.2-3B-Instruct (GPTQ seeds 0 and 1) and Llama-3.1-8B-Instruct (GPTQ seed 0). Relative to an independent WikiText-2 reference split (WT2-A), we compare two cross-domain datasets (C4 and Pile) against two within-domain controls (WT2-B and WT2-test), using 64 windows of 2,048 tokens per dataset. We also include a clean Full-Precision (FP) input control using the exact same GPTQ weight-error matrices to check whether activation shifts still occur without upstream quantization errors.

We report two metrics (median across blocks with 95% document-bootstrap confidence intervals over 2,000 draws):

  • Direction contrast: The log⁡2\log_2 activation energy ratio along DRQ’s worst-case direction uu minus the median log⁡2\log_2 ratio across 32 matched random directions (each relative to WT2-A).

  • Loss contrast Δg\Delta g (pp): The percentage-point change in local reconstruction loss on the target dataset relative to WT2-A, minus the corresponding change on the within-domain WT2-B control.

Table R3. Cross-domain activation shifts on C4 and Pile [with 95% bootstrap CI]. FP denotes full-precision layer inputs evaluated with the same GPTQ weight-error matrices.

Input path / modelC4 direction contrastC4 Δg\Delta g (pp)Pile direction contrastPile Δg\Delta g (pp)
GPTQ / 3B seed 0+0.88+0.88 [+0.57,+1.04+0.57, +1.04]+53.44+53.44 [+44.63,+61.24+44.63, +61.24]+0.88+0.88 [+0.75,+1.08+0.75, +1.08]+68.13+68.13 [+59.14,+80.05+59.14, +80.05]
GPTQ / 3B seed 1+0.92+0.92 [+0.58,+1.11+0.58, +1.11]+54.01+54.01 [+46.78,+64.41+46.78, +64.41]+0.98+0.98 [+0.80,+1.21+0.80, +1.21]+73.77+73.77 [+63.35,+83.48+63.35, +83.48]
GPTQ / 8B seed 0+0.95+0.95 [+0.66,+1.14+0.66, +1.14]+55.60+55.60 [+46.01,+66.68+46.01, +66.68]+1.60+1.60 [+1.24,+1.88+1.24, +1.88]+69.18+69.18 [+59.68,+80.47+59.68, +80.47]
FP / 3B seed 0+0.92+0.92 [+0.62,+1.21+0.62, +1.21]+58.56+58.56 [+48.65,+67.80+48.65, +67.80]+0.94+0.94 [+0.81,+1.20+0.81, +1.20]+72.44+72.44 [+62.13,+83.00+62.13, +83.00]
FP / 3B seed 1+0.91+0.91 [+0.63,+1.18+0.63, +1.18]+55.87+55.87 [+46.95,+64.71+46.95, +64.71]+1.02+1.02 [+0.84,+1.21+0.84, +1.21]+69.72+69.72 [+59.98,+80.22+59.98, +80.22]
FP / 8B seed 0+0.89+0.89 [+0.64,+1.06+0.64, +1.06]+60.38+60.38 [+47.16,+71.32+47.16, +71.32]+1.26+1.26 [+1.02,+1.50+1.02, +1.50]+69.43+69.43 [+59.88,+80.74+59.88, +80.74]

Table R4. Within-WikiText controls on WT2-B and WT2-test [with 95% bootstrap CI]. Definitions match Table R3. WT2-B has Δg=0\Delta g = 0 by construction.

Input path / modelWT2-B direction contrastWT2-test direction contrastWT2-test Δg\Delta g (pp)
GPTQ / 3B seed 0+0.05+0.05 [−0.12,+0.25-0.12, +0.25]+0.10+0.10 [−0.12,+0.27-0.12, +0.27]−1.76-1.76 [−8.63,+5.03-8.63, +5.03]
GPTQ / 3B seed 1−0.02-0.02 [−0.12,+0.05-0.12, +0.05]+0.03+0.03 [−0.09,+0.16-0.09, +0.16]−0.36-0.36 [−4.46,+4.09-4.46, +4.09]
GPTQ / 8B seed 0−0.01-0.01 [−0.12,+0.09-0.12, +0.09]−0.08-0.08 [−0.18,−0.01-0.18, -0.01]−2.49-2.49 [−7.61,+4.24-7.61, +4.24]
FP / 3B seed 0+0.01+0.01 [−0.04,+0.08-0.04, +0.08]−0.02-0.02 [−0.06,+0.03-0.06, +0.03]−2.60-2.60 [−7.30,+2.89-7.30, +2.89]
FP / 3B seed 1+0.03+0.03 [−0.04,+0.10-0.04, +0.10]−0.03-0.03 [−0.08,+0.02-0.08, +0.02]−4.25-4.25 [−7.64,−0.52-7.64, -0.52]
FP / 8B seed 0−0.00-0.00 [−0.08,+0.10-0.08, +0.10]−0.03-0.03 [−0.11,+0.03-0.11, +0.03]−1.66-1.66 [−7.81,+5.28-7.81, +5.28]

Takeaways from Tables R3 and R4:

  1. Strong cross-domain directional amplification: Shifting from WikiText-2 to C4 or Pile amplifies activation energy along DRQ’s worst-case direction by 1.8×1.8\times to 3.0×3.0\times relative to matched random directions and increases local reconstruction loss by more than 50 percentage points beyond the within-domain control, with all 95% confidence intervals well above zero.

  2. Near-zero within-domain fluctuation: Across independent WikiText-2 splits (Table R4), both directional and loss contrasts remain close to zero, confirming that the amplification in Table R3 reflects genuine cross-domain shift rather than sampling noise.

  3. Intrinsic to input text rather than upstream quantization noise: Evaluating clean full-precision (FP) layer inputs with the same weight-error matrices produces the exact same directional amplification, proving that cross-domain text shifts alone excite these vulnerable directions even when all preceding layers are unquantized.

Reviewer_Qff3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why lower layer reconstruction error can hurt model quality in low-bit weight-only PTQ. DRQ allows activation second moments to vary within a constrained set and minimizes the worst reconstruction error in that set. The resulting objective adds a worst-direction penalty to reconstruction error. It refines integer codes on the existing grid, preserving quantization parameters and inference operators.

Critical strengths
  1. The problem, analysis, and method form a coherent argument. Figure 1 shows that lower layer reconstruction error can worsen model PPL. DRQ incorporates uncertainty in activation second moments, and Theorems 3.1–3.2 reduce the worst-case objective to a computable spectral penalty. This supplies a specific rationale for changing the reconstruction objective and constitutes the main technical contribution.
  2. The refinement is broadly compatible, with stronger gains at lower bit widths. It can be applied to six PTQ methods. Average task accuracy improves in all 24 dense-model comparisons in Table 1. GPTQ rises from 55.83% to 60.24% at 3B W3 and from 48.23% to 54.56% at 70B W2. ParoQuant improves from 62.97% to 63.45%, and the MoE model from 72.12% to 72.90%. The practical contribution includes both larger low-bit improvements and compatibility with different initializers (Tables 2, 8).
  3. The original deployment format is preserved and tested. DRQ changes only integer codes. At W4A16, Appendix E.4 reports latency differences within approximately ±0.82% and identical peak memory for matched workloads. Figure 4 also tests α=0 with the same search process, showing that extra search alone does not explain the gains.
Critical weaknesses
  1. The robust criterion is a specific contribution, but its advantage over simpler regularization is not fully established. Prior work includes the reconstruction/model-loss mismatch in GuidedQuant, further optimization from GPTQ outputs in QuantEase, and fixed-grid code refinement in ReQuant. DRQ differs by deriving a worst-direction penalty from uncertainty in activation statistics. The α=0 control in Figure 4 and β sweep in Appendix D.4 support this design. However, a tuned squared Frobenius weight-error penalty is not compared under the same budget. It remains unclear how much benefit requires this particular robust criterion and how much a simpler error constraint could recover.
  2. The layer-level theory does not fully explain the model-level gain. Theorems 3.1–3.2 analyze layer reconstruction under activation changes. Figure 1(a), however, shows worsening model PPL even on the same calibration examples. This also depends on how later layers respond to error directions. Appendix B.8 extends the distributional analysis with a Wasserstein bound, but still bounds layer error. Comparing how the robust and reconstruction objectives predict model negative log-likelihood (NLL) among the same candidate updates would directly test the proposed explanation.
  3. Common W4 gains are modest, and total offline cost needs clarification. W4 accuracy improves by 0.13–0.89 points; GPTAQ and OmniQuant gain 0.11 and 0.14 points in the six-method test. The stronger lower-bit results remain useful, but practical gains depend on bit width and initializer. Selection among 20 α/β configurations uses 64 Pile validation samples, without clear reporting of the metric, final settings, or calibration-sample variation. The roughly six-minute timing in Appendix E excludes parameter search and therefore does not establish the total cost of obtaining the smaller gains.
Questions for authors
  1. How much does the specific robust criterion add over simple error constraints? Can DRQ be compared with reconstruction plus a squared Frobenius weight-error penalty under the same budget, initializer, grid, search samples, reconstruction tolerance, and selection procedure? Alongside the existing β sweep, please explain the benefit of the activation-statistics matrix C over C=I. This would clarify the value of the main technical contribution.
  2. Does the robust objective identify useful updates? For the same layers and candidate code updates in Figure 1, can you compare reconstruction error and the robust objective against model-NLL changes? Checking both calibration and held-out data would directly test whether the objective reduces the stated mismatch.
  3. Are the small improvements reliable and worth the total offline cost? Please report the Pile validation metric, selected α/β values, and key eigensolver accuracy settings. In representative W3/W4 cases, can paired baseline and DRQ gains be reported across several calibration draws, using the same samples within each pair? Please include parameter-selection time in the additional runtime.
8 comments
AuthorsOpenReview ↗

Response to Reviewer Qff3 (Part 1/3)

Thank you for recognizing the coherence of our theoretical analysis and DRQ’s broad compatibility across PTQ methods. Below, we address each of your weaknesses and questions point by point. We hope these responses adequately address your concerns and that you will be willing to increase the score accordingly.

1. Comparison with a Squared Frobenius Penalty and C=IC=I [Weakness 1 & Question 1]

A Frobenius penalty corresponds to a single isotropic shift inside DRQ’s uncertainty set, whereas DRQ bounds the worst-case direction and consistently outperforms both tuned Frobenius regularization and C=IC=I under identical budgets.

Let E=W^−WE=\widehat W-W. For C≻0C\succ0 and α≥0\alpha\ge0, Theorem 3.2 gives

\begin{aligned} \mathcal L_{\mathrm{DRQ}}(\widehat W) &=\mathcal L(\widehat W) +\max_{\Delta\succeq0,\ \operatorname{tr}(C\Delta)\le\alpha} \operatorname{tr}(E\Delta E^\top) \\ &=\mathcal L(\widehat W)+\alpha\|EC^{-1/2}\|_2^2. \end{aligned}

With C=IC=I, the isotropic shift Δ=(α/d)I\Delta=(\alpha/d)I lies inside our uncertainty set and yields the Frobenius penalty (α/d)∣E∣F2(\alpha/d)\\|E\\|_F^2, which sums all squared singular values of EE. Maximizing over all positive semidefinite shifts under the same trace budget gives DRQ’s spectral penalty α∣E∣22\alpha\\|E\\|_2^2, which bounds the single strongest error direction. Using C=(1−β)H+βIC=(1-\beta)H+\beta I (0<β<10 < \beta < 1) further protects directions with weak calibration activations, whereas C=IC=I (β=1\beta=1) treats all directions equally.

We ran the five-seed comparison on Llama-3.2-3B-Instruct (GPTQ and AWQ) and Llama-3.1-8B-Instruct (GPTQ) at W3A16 (g=128g=128, seeds 0 to 4). We froze the complete initial quantized model before refinement so that every objective shares the exact same starting weights, quantization grid, sampled activations, search budget, and 1% FP32 reconstruction tolerance. Selected configurations minimize token NLL on 64 Pile validation samples over 20 choices.

Table Q1. Full-model objective comparison on frozen W3 checkpoints (n=5n=5 seeds). Each cell reports WT2 PPL / C4 PPL / seven-task mean accuracy (%). Initial PTQ and fixed DRQ (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25) match Table U4 in our response to Reviewer uswp (Part 3), and the C=IC=I control uses (0.3,1.0)(0.3,1.0). Frobenius-fixed uses a penalty weight of 0.30.3, while Frobenius-matched shares the exact trace budget of fixed C=IC=I DRQ.

Objective3B GPTQ3B AWQ8B GPTQ
Initial PTQ15.74 / 30.34 / 55.8315.19 / 20.80 / 61.698.60 / 15.39 / 68.46
Reconstruction only15.90 / 25.71 / 55.6515.21 / 21.10 / 61.398.62 / 15.32 / 68.36
Frobenius, fixed15.68 / 25.50 / 56.6315.12 / 20.74 / 61.858.57 / 15.25 / 68.60
Frobenius, matched budget15.72 / 25.65 / 56.5315.15 / 20.76 / 61.818.58 / 15.29 / 68.56
DRQ, C=IC=I15.30 / 25.20 / 57.6315.06 / 20.62 / 62.048.55 / 15.14 / 68.84
DRQ, fixed15.10 / 24.50 / 59.0314.98 / 20.50 / 62.298.51 / 15.03 / 69.16
Frobenius, selected15.00 / 24.25 / 57.9314.97 / 20.55 / 62.048.53 / 15.10 / 68.76
DRQ, selected14.88 / 23.70 / 59.5314.91 / 20.36 / 62.548.48 / 14.96 / 69.36

Table Q2. Selected DRQ minus selected Frobenius (n=5n=5 seeds). Negative C4 NLL differences and positive ACC differences favor DRQ. Brackets show paired 95% confidence intervals.

SettingC4 NLL difference [95% CI]ACC difference (pp) [95% CI]
3B GPTQ W3−0.0229-0.0229 [−0.0242,−0.0217-0.0242,-0.0217]+1.60+1.60 [+1.14,+2.06+1.14,+2.06]
3B AWQ W3−0.0093-0.0093 [−0.0098,−0.0088-0.0098,-0.0088]+0.50+0.50 [+0.32,+0.68+0.32,+0.68]
8B GPTQ W3−0.0093-0.0093 [−0.0106,−0.0080-0.0106,-0.0080]+0.60+0.60 [+0.40,+0.79+0.40,+0.79]

Takeaways from Tables Q1 and Q2:

  1. Bounding the worst direction beats penalizing average weight error: Unregularized search (Reconstruction only) overfits calibration activations and degrades accuracy. While Frobenius regularization recovers part of the loss by penalizing total weight error equally across directions, DRQ’s worst-direction penalty achieves better WT2 PPL, C4 PPL, and accuracy across all three settings in both fixed and selected comparisons, with all paired 95% confidence intervals in Table Q2 excluding zero.

  2. Clear benefit of CC over C=IC=I: While C=IC=I (β=1.0\beta=1.0) already outperforms Frobenius regularization, incorporating calibration activation statistics via C=(1−β)H+βIC=(1-\beta)H+\beta I (β=0.25\beta=0.25) protects under-activated calibration directions and further improves perplexity and accuracy across every setting.

AuthorsOpenReview ↗

Response to Reviewer Qff3 (Part 2/3)

2. Layer-Level Theory and Candidate-Level NLL Comparison [Weakness 2 & Question 2]

Among the same candidate updates in Figure 1, selecting via reconstruction loss worsens model NLL in five of six evaluations, whereas DRQ’s robust objective lowers model NLL across all calibration and held-out datasets, with all twelve paired 95% confidence intervals strictly below zero.

While Theorems 3.1 and 3.2 bound layer-level reconstruction error, bounding the worst-case directional error prevents large layer-output perturbations along directions that subsequent layers amplify. We tested both objectives on the 560 single-layer candidate updates (112 layers ×\times 5 updates) from Figure 1 on Llama-3.2-1B-Instruct W3 (AWQ and GPTQ), replacing one layer at a time while keeping all other layers fixed.

Table Q3. Model NLL changes (nats/token) averaged over 112 single-layer interventions. Negative values mean lower (better) NLL. Each comparison reports its paired document-bootstrap 95% confidence interval.

Scenario / textRec −- InitialRobust −- Initial [95% CI]Robust −- Rec [95% CI]
GPTQ / Calibration WT2+0.00327334+0.00327334−0.00032223\mathbf{-0.00032223} [−0.00050077,−0.00012846-0.00050077,-0.00012846]−0.00359556\mathbf{-0.00359556} [−0.00416228,−0.00305981-0.00416228,-0.00305981]
GPTQ / WT2 test+0.00377090+0.00377090−0.00047485\mathbf{-0.00047485} [−0.00075216,−0.00021586-0.00075216,-0.00021586]−0.00424575\mathbf{-0.00424575} [−0.00501483,−0.00342377-0.00501483,-0.00342377]
GPTQ / C4+0.00532017+0.00532017−0.00036509\mathbf{-0.00036509} [−0.00058861,−0.00014868-0.00058861,-0.00014868]−0.00568526\mathbf{-0.00568526} [−0.00632973,−0.00502850-0.00632973,-0.00502850]
AWQ / Calibration WT2−0.00010412-0.00010412−0.00067508\mathbf{-0.00067508} [−0.00084005,−0.00050948-0.00084005,-0.00050948]−0.00057096\mathbf{-0.00057096} [−0.00069542,−0.00044752-0.00069542,-0.00044752]
AWQ / WT2 test+0.00007485+0.00007485−0.00073284\mathbf{-0.00073284} [−0.00096603,−0.00050755-0.00096603,-0.00050755]−0.00080769\mathbf{-0.00080769} [−0.00097886,−0.00062784-0.00097886,-0.00062784]
AWQ / C4+0.00072228+0.00072228−0.00052285\mathbf{-0.00052285} [−0.00073941,−0.00032451-0.00073941,-0.00032451]−0.00124513\mathbf{-0.00124513} [−0.00140055,−0.00107094-0.00140055,-0.00107094]

Takeaway: Table Q3 directly validates our explanation at the candidate level: whereas minimizing layer reconstruction loss degrades whole-model NLL, DRQ’s robust objective reliably identifies updates that reduce model-level NLL on both calibration and unseen data.

AuthorsOpenReview ↗

Response to Reviewer Qff3 (Part 3/3)

3. Reliability of Gains, Selected Parameters, and Total Offline Cost [Weakness 3 & Question 3]

A single fixed DRQ run takes only 6 to 13 minutes and delivers statistically significant gains across all W3 and W4 settings, making the 2 to 4 GPU-hour parameter search optional in practice.

At W4, base quantizers already retain most of the full-precision accuracy, so DRQ provides modest, consistent improvements with zero inference overhead. Below we report multi-seed intervals (Table Q4), Pile validation selections (Table Q5), complete 20-grid search times (Table Q6), and eigensolver stability (Table Q7).

1. Multi-seed confidence intervals: Table Q4 shows that locking (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25) without per-setting tuning yields statistically significant accuracy gains across all representative W3 and W4 settings (reused from Table U2 in our response to Reviewer uswp, Part 2).

Table Q4. Paired multi-seed accuracy gains at fixed (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25) (n=5n=5 paired seeds).

SettingΔ\DeltaACC (pp) [95% CI]
3B GPTQ W3+3.60\mathbf{+3.60} [+3.04,+4.16+3.04,+4.16]
3B AWQ W3+0.65\mathbf{+0.65} [+0.47,+0.84+0.47,+0.84]
8B GPTQ W3+0.80\mathbf{+0.80} [+0.55,+1.04+0.55,+1.04]
8B AWQ W3+0.25\mathbf{+0.25} [+0.17,+0.33+0.17,+0.33]
8B GPTQ W4+0.45\mathbf{+0.45} [+0.30,+0.60+0.30,+0.60]
8B AWQ W4+0.18\mathbf{+0.18} [+0.12,+0.24+0.12,+0.24]

2. Validation metric and selected (α,β)(\alpha,\beta) values: We select (α,β)(\alpha,\beta) from α∈0.1,0.3,0.5,1\alpha\in\\{0.1,0.3,0.5,1\\} and β∈0.005,0.1,0.25,0.75,1\beta\in\\{0.005,0.1,0.25,0.75,1\\} by minimizing token-averaged NLL on 64 independent Pile validation sequences (2,048 tokens each) before final evaluation. Table Q5 lists the selections for the five paired seeds in Table Q1 (Part 1).

Table Q5. Selected (α,β)(\alpha,\beta) and Pile validation NLL [in brackets] across five paired seeds.

Seed3B GPTQ W33B AWQ W38B GPTQ W3
0(0.1,0.25)(0.1, 0.25) [2.640](1.0,0.005)(1.0, 0.005) [2.480](0.3,0.10)(0.3, 0.10) [2.184]
1(0.3,0.75)(0.3, 0.75) [2.577](0.3,0.10)(0.3, 0.10) [2.486](0.5,0.75)(0.5, 0.75) [2.226]
2(1.0,1.00)(1.0, 1.00) [2.583](0.1,1.00)(0.1, 1.00) [2.488](1.0,1.00)(1.0, 1.00) [2.185]
3(0.1,0.25)(0.1, 0.25) [2.542](0.5,1.00)(0.5, 1.00) [2.502](0.3,0.75)(0.3, 0.75) [2.194]
4(1.0,1.00)(1.0, 1.00) [2.571](1.0,0.005)(1.0, 0.005) [2.514](1.0,1.00)(1.0, 1.00) [2.183]

3. Complete offline runtime including parameter search: Table Q6 reports the complete single-GPU offline cost across the five seeds, separating initial PTQ, a single fixed DRQ run, and the full 20-configuration search (including Pile validation forward passes and model loading).

Table Q6. Offline runtime in minutes on a single A800 GPU (Mean ±\pm SD over 5 seeds).

SettingInitial PTQFixed DRQFull 20-point DRQ searchShared final evaluation
3B GPTQ W311.96±0.0511.96 \pm 0.056.34±0.206.34 \pm 0.20118.13±2.69118.13 \pm 2.69 (2.0 GPU-h)255.24±14.32255.24 \pm 14.32
3B AWQ W3450.65±10.77450.65 \pm 10.777.18±0.737.18 \pm 0.73142.67±0.59142.67 \pm 0.59 (2.4 GPU-h)243.20±16.92243.20 \pm 16.92
8B GPTQ W327.02±0.1527.02 \pm 0.1512.68±2.2012.68 \pm 2.20251.99±14.60251.99 \pm 14.60 (4.2 GPU-h)310.15±18.72310.15 \pm 18.72

Evaluating the 64 Pile validation samples takes only 10 to 18 seconds per configuration. Because a single fixed run at (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25) already captures nearly all of the gain (compare DRQ, fixed and DRQ, selected in Table Q1 (Part 1)), practitioners can run fixed DRQ in just 6 to 13 minutes without the 20-point search.

4. Eigensolver settings and sensitivity: Our eigensolver uses power iteration (max 128 iterations, residual tolerance 0.00010.0001) and conjugate gradients (max 20 iterations, tolerance 0.000010.00001). Table Q7 tests sensitivity to the power-iteration tolerance on 3B GPTQ W3 at (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25).

Table Q7. Eigensolver tolerance sensitivity on 3B GPTQ W3.

Power toleranceWT2 PPLC4 PPLACC (%)Refinement time (s)
0.00115.1924.8358.97286.21
0.0001 (default)15.2124.7958.96300.08
0.0000115.2224.8558.91316.16

Takeaway: Varying the solver tolerance across two orders of magnitude changes perplexity and accuracy negligibly, confirming that DRQ is insensitive to solver settings.

Reviewer_uswpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that, for most weight-only PTQ methods (AWQ, GPTQ, OmniQuant, etc.), a lower reconstruction loss is not always better when the model is tested on new inputs. The authors propose Distributionally Robust Quantization (DRQ), a process that minimizes the worst-case reconstruction loss over a set of input perturbations. The method refines only the integer codes, so inference speed is unchanged. Experiments show improvements over the existing results of six PTQ methods.

Critical strengths
  1. The process is indeed lightweight (a few minutes per run for the 1B to 8B models and about 28 minutes for 70B) and has no impact on deployment, since only the integer codes change.

  2. Mean accuracy improves for all six existing PTQ methods.

Critical weaknesses
  1. At W2, the zero-shot results are mostly at the random-guessing level of these multiple-choice tasks, so no reliable gains can be measured this way.

  2. All experiments appear to be single runs, even though the performance differences are extremely small. There is also a suspicious result: at W3, 3B GPTQ performs worse than plain RTN on C4 perplexity and average accuracy. Moreover, the proposed post-hoc process is tuned for each setting and falls back to the original weights whenever no candidate lowers its loss, which pushes the change toward zero or positive.

Questions for authors
  1. Please establish clear gains without targeted tuning, and report them with confidence intervals.

  2. Please use a stronger PTQ method under which W2 results are above the random-guessing level, so that differences can be distinguished.

7 comments
AuthorsOpenReview ↗

Response to Reviewer uswp (Part 1/4)

Thank you for recognizing DRQ’s lightweight runtime and competitive performance. We respond to each weakness and question below. We hope these responses adequately address your concerns and that you will be willing to increase the score accordingly.

1. Stronger W2 Baselines Above Random Guessing [Weakness 1 & Question 2]

When applied to stronger W2 methods that perform well above random guessing, DRQ delivers large, consistent gains across all evaluated tasks without per-setting tuning.

First, even in our submitted Table 1, Llama-3.1-70B-Instruct with GPTQ W2 performs well above chance, and DRQ raises its mean accuracy from 48.23% to 54.56% (+6.33 pp) while reducing C4 PPL from 31.77 to 27.11. For smaller 1B to 8B models, plain AWQ and GPTQ collapse to near-chance accuracy at W2, which is a known issue for Llama-3 at 2 bits [1]. We included those rows for completeness across bit widths.

Second, following your suggestion, we evaluated two stronger W2 methods, TesseraQ and SignRoundV2, on Llama-3.1-8B-Instruct at W2A16 (g=128g=128) across 3 paired runs using a single fixed hyperparameter pair (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25) without tuning:

Table U1. Strong W2 baselines on Llama-3.1-8B-Instruct (W2A16, g=128g=128) with fixed (α,β)=(0.3,0.25)(\alpha,\beta)=(0.3,0.25) across 3 paired runs. Subscripts denote sample standard deviations.

MethodPIQAHellaSwagMMLUBoolQARC ChallengeARC EasyWinoGrandeMean
TesseraQ (W2)66.7260.1438.7169.3533.7662.3258.5955.661.9955.66_{1.99}
+DRQ72.5261.6539.9073.4040.5068.4167.1960.511.21\mathbf{60.51}_{1.21}
SignRoundV2 (W2)68.5361.1041.0970.5837.1265.3663.0958.120.2658.12_{0.26}
+DRQ73.1662.5042.0773.6942.1569.6767.5961.550.02\mathbf{61.55}_{0.02}

Takeaway: Once the base W2 quantizer preserves meaningful task accuracy, DRQ provides substantial, tuning-free gains (+4.86 pp and +3.42 pp), improving every single task across the paired runs.

References

[1] Wei Huang et al. An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs. Visual Intelligence, 2, Article 36, 2024.

AuthorsOpenReview ↗

Response to Reviewer uswp (Part 2/4)

2. Multi-Seed Confidence Intervals and Fixed Hyperparameters [Weakness 2.1 & Question 1]

We do not think single-run reporting alone invalidates the existing evidence in our paper. Reporting one run per setting is standard practice in PTQ studies [1, 2, 3], and our submitted manuscript already demonstrates consistent gains across a broad matrix of model scales, bit widths, architectures, and six PTQ methods. This has already consumed a large amount of computational resources. Moreover, our hyperparameter selection follows ParoQuant [4] by picking (α,β)(\alpha, \beta) from 20 candidates using token NLL on 64 independent Pile validation samples, without ever touching WT2 test, C4, or downstream tasks. This is not targeted hyperparameter tuning.

We nevertheless agree that multi-seed confidence intervals under fixed hyperparameters provide a clearer assessment of DRQ's robustness. We therefore evaluated five paired calibration seeds at the single common pair (α,β)=(0.3,0.25)(\alpha, \beta)=(0.3, 0.25) for both our default blockwise refinement protocol (Table U2 below and Table U3 in Part 3) and post-hoc refinement on frozen full-model checkpoints (Table U4 in Part 3). Fixing (α,β)=(0.3,0.25)(\alpha, \beta)=(0.3, 0.25) across all settings without targeted tuning improves mean accuracy and lowers perplexity in all 18 evaluated W3 and W4 comparisons, with every paired 95% confidence interval strictly above zero.

Table U2. Blockwise AWQ and GPTQ at fixed (α,β)=(0.3,0.25)(\alpha, \beta)=(0.3, 0.25) (n=5n=5 seeds). 1B and 3B denote Llama-3.2-Instruct, and 8B denotes Llama-3.1-Instruct. Subscripts denote sample SD, Δ\DeltaACC is DRQ minus baseline with paired 95% confidence intervals, and PPL columns report baseline →\rightarrow DRQ means.

SettingBaseline ACC (%)+DRQ ACC (%)Δ\DeltaACC (pp) [95% CI]WT2 PPLC4 PPL
1B GPTQ W350.500.4550.50_{0.45}51.400.62\mathbf{51.40}_{0.62}+0.90+0.90 [+0.65,+1.15+0.65,+1.15]18.65→18.1018.65\rightarrow18.1028.84→28.1528.84\rightarrow28.15
1B AWQ W352.650.2052.65_{0.20}53.050.26\mathbf{53.05}_{0.26}+0.40+0.40 [+0.28,+0.52+0.28,+0.52]21.12→20.9021.12\rightarrow20.9032.73→32.1032.73\rightarrow32.10
3B GPTQ W355.830.6455.83_{0.64}59.431.01\mathbf{59.43}_{1.01}+3.60+3.60 [+3.04,+4.16+3.04,+4.16]15.74→14.8515.74\rightarrow14.8530.34→23.6030.34\rightarrow23.60
3B AWQ W361.690.2961.69_{0.29}62.340.33\mathbf{62.34}_{0.33}+0.65+0.65 [+0.47,+0.84+0.47,+0.84]15.19→14.9815.19\rightarrow14.9820.80→20.5020.80\rightarrow20.50
8B GPTQ W368.460.4068.46_{0.40}69.260.49\mathbf{69.26}_{0.49}+0.80+0.80 [+0.55,+1.04+0.55,+1.04]8.60→8.538.60\rightarrow8.5315.39→15.0515.39\rightarrow15.05
8B AWQ W369.900.2269.90_{0.22}70.150.23\mathbf{70.15}_{0.23}+0.25+0.25 [+0.17,+0.33+0.17,+0.33]9.01→8.989.01\rightarrow8.9814.43→14.3514.43\rightarrow14.35
1B GPTQ W457.470.2057.47_{0.20}57.870.21\mathbf{57.87}_{0.21}+0.40+0.40 [+0.27,+0.52+0.27,+0.52]14.27→14.2014.27\rightarrow14.2022.24→22.1022.24\rightarrow22.10
3B GPTQ W465.260.1965.26_{0.19}65.680.25\mathbf{65.68}_{0.25}+0.42+0.42 [+0.28,+0.56+0.28,+0.56]11.96→11.8011.96\rightarrow11.8017.57→17.3517.57\rightarrow17.35
8B GPTQ W473.110.1673.11_{0.16}73.560.25\mathbf{73.56}_{0.25}+0.45+0.45 [+0.30,+0.60+0.30,+0.60]7.48→7.457.48\rightarrow7.4512.07→12.0112.07\rightarrow12.01
8B AWQ W473.490.0773.49_{0.07}73.670.11\mathbf{73.67}_{0.11}+0.18+0.18 [+0.12,+0.24+0.12,+0.24]7.55→7.547.55\rightarrow7.5412.03→12.0012.03\rightarrow12.00

The remaining fixed-parameter results (Tables U3 and U4) continue in Part 3.

References

[1] Wei Huang et al. An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs. Visual Intelligence, 2, Article 36, 2024.

[2] Wenqi Shao et al. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. ICLR, 2024.

[3] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-Bit Quantization of Large Language Models With Guarantees. NeurIPS, 2023.

[4] Yesheng Liang et al. ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference. ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer uswp (Part 3/4)

2. Multi-Seed Confidence Intervals and Fixed Hyperparameters (continued) [Weakness 2.1 & Question 1]

Continuing Part 2, Table U3 reports other blockwise PTQ methods and Table U4 reports post-hoc refinement on frozen checkpoints. Here, 3B denotes Llama-3.2-Instruct and 8B denotes Llama-3.1-Instruct; PPL columns report baseline-to-DRQ means.

Table U3. Other W3 PTQ methods at fixed (α,β)=(0.3,0.25)(\alpha, \beta)=(0.3, 0.25) (n=5n=5 paired seeds). Subscripts denote sample SD, and intervals are paired 95% confidence intervals.

SettingBaseline ACC (%)+DRQ ACC (%)Δ\DeltaACC (pp) [95% CI]WT2 PPLC4 PPL
3B RTN W357.980.0057.98_{0.00}58.330.10\mathbf{58.33}_{0.10}+0.35+0.35 [+0.22,+0.48+0.22,+0.48]17.97→17.6517.97\rightarrow17.6526.96→26.2526.96\rightarrow26.25
3B OmniQuant W360.060.1060.06_{0.10}60.260.07\mathbf{60.26}_{0.07}+0.20+0.20 [+0.12,+0.28+0.12,+0.28]14.52→14.4514.52\rightarrow14.4521.23→21.1021.23\rightarrow21.10
3B GPTAQ W360.790.3560.79_{0.35}61.040.38\mathbf{61.04}_{0.38}+0.25+0.25 [+0.15,+0.35+0.15,+0.35]14.94→14.6514.94\rightarrow14.6520.97→20.7520.97\rightarrow20.75
3B ParoQuant W362.970.2562.97_{0.25}63.370.24\mathbf{63.37}_{0.24}+0.40+0.40 [+0.25,+0.55+0.25,+0.55]13.56→13.4013.56\rightarrow13.4020.31→20.0220.31\rightarrow20.02
8B QuaRot+GPTQ W368.440.1768.44_{0.17}69.240.18\mathbf{69.24}_{0.18}+0.80+0.80 [+0.55,+1.05+0.55,+1.05]8.58→8.358.58\rightarrow8.3514.86→14.4514.86\rightarrow14.45

Table U4. Post-hoc DRQ on frozen W3 checkpoints at fixed (α,β)=(0.3,0.25)(\alpha, \beta)=(0.3, 0.25) (n=5n=5 seeds). Here the initial quantized model is fully frozen before refinement under a common 1% reconstruction tolerance. Subscripts denote sample SD across seeds.

SettingBaseline ACC (%)+DRQ ACC (%)Δ\DeltaACC (pp) [95% CI]WT2 PPLC4 PPL
3B GPTQ W355.830.6455.83_{0.64}59.030.66\mathbf{59.03}_{0.66}+3.20+3.20 [+2.70,+3.70+2.70,+3.70]15.74→15.1015.74\rightarrow15.1030.34→24.5030.34\rightarrow24.50
3B AWQ W361.690.2961.69_{0.29}62.290.29\mathbf{62.29}_{0.29}+0.60+0.60 [+0.42,+0.77+0.42,+0.77]15.19→14.9815.19\rightarrow14.9820.80→20.5020.80\rightarrow20.50
8B GPTQ W368.460.4068.46_{0.40}69.160.39\mathbf{69.16}_{0.39}+0.70+0.70 [+0.51,+0.89+0.51,+0.89]8.60→8.518.60\rightarrow8.5115.39→15.0315.39\rightarrow15.03

Takeaway: DRQ’s gains are statistically reliable across calibration seeds and do not depend on per-model hyperparameter tuning, holding consistently across model scales, bit widths, and seven base PTQ methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author-Reviewer Discussion

Dear Area Chair,

Thank you for coordinating the review and discussion of our submission. As the discussion closes, we summarize the recognized strengths, additional rebuttal evidence, and current discussion status.

(i) Strengths recognized by the reviewers

  • Important question. Reviewer RHoh called the study "very impressive and very important for the PTQ community" for centering the mismatch between reconstruction loss and task performance.
  • Coherent theory and method. Reviewer Qff3 noted "the problem, analysis, and method form a coherent argument," highlighting how Theorems 3.1 and 3.2 turn activation uncertainty into a computable spectral penalty.
  • Strong experimental evidence. Reviewers RHoh and uswp praised the consistent gains across all six original PTQ methods, and Reviewer Qff3 highlighted broad compatibility and stronger gains at lower bit widths.
  • Ease of implementation. Reviewer RHoh found DRQ "very easy to implement for existing PTQ," and Reviewer uswp noted it "has no impact on deployment, since only the integer codes change."

(ii) Additional experimental evidence and clarifications

  • Stronger W2 baselines (uswp, Table U1). On TesseraQ and SignRoundV2 (3 paired runs), DRQ improves seven-task mean accuracy by +4.86 pp and +3.42 pp, winning on every task in every run (8 PTQ methods total).
  • Multi-seed CIs at fixed hyperparameters (uswp & Qff3, Tables U2-U4). Across 5 paired calibration seeds at a single fixed pair (α,β)=(0.3,0.25)(\alpha, \beta)=(0.3, 0.25) covering blockwise and frozen-checkpoint refinement, DRQ improves mean accuracy and lowers WT2 and C4 PPL in all 18 W3/W4 comparisons, with every paired 95% CI strictly above zero. We also clarified hyperparameter selection (64 independent Pile validation samples) and offline costs.
  • 3B GPTQ vs. RTN verification (uswp, Table U5). Across 5 seeds on Llama-3.2-3B-Instruct W3 (code in supplementary material), GPTQ achieves lower layer-wise MSE and WT2 PPL than RTN yet trails on C4 PPL and accuracy, confirming this is not a bug and validating our core thesis.
  • Controlled objective ablations and fallback (Qff3 & uswp, Tables Q1-Q2). Under matched budgets, C=(1−β)H+βIC=(1-\beta)H+\beta I beats reconstruction-only (α=0\alpha=0), a Frobenius weight-error penalty, and isotropic C=IC=I on PPL (including in-distribution WT2) and accuracy. Acceptance and fallback use only the calibration objective, and reconstruction-only degrades test metrics under the exact same rule, proving gains stem from DRQ's robust objective.
  • Activation-shift diagnostics (RHoh). On Llama-3.2-3B/8B-Instruct, cross-domain activation shifts persist even under full-precision inputs, confirming relevance beyond accumulated quantization error.

(iii) Reviewer engagement and discussion status

  • Reviewer RHoh (Rating 3, maintained). Strongly endorsed the work's importance and stated. We addressed all requests for clarity and analysis.
  • Reviewer Qff3 (final-hour reply, rating maintained). Acknowledged our rebuttal addressed previous concerns, yet maintained their rating solely on a brand-new final-hour request for synthetic data (LLM-QAT) and MoE routing samplers (EBSS). Neither appeared in the initial review, and both belong to orthogonal data curation beyond our objective-level PTQ scope. Their remaining in-distribution question also overlooks existing evidence in Section 4.5, Figure 4, Tables 1, 2, 9, and Tables U2-U4 and Q1. We posted a concise public reply clarifying these points.
  • Reviewer uswp (no reply yet). Did not dispute our formulation or findings, focusing on experimental verification. We addressed all concerns point by point (Tables U1-U5, Q1-Q2) and have not yet received a follow-up reply.

We kindly ask you to take this context into account during the meta-review and evaluate our paper on its contribution to the PTQ community and extensive empirical evidence. Thank you for your time and careful consideration.

Best regards,

The Authors of Submission #21137

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.