- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
"When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization" identifies the schematic problem of PTQ that minimization of the reconstruction loss per layer does not necssarliry means the minimization of the task loss. This paper mathematically shows when this discrepancy occurs and identifies that non-active directions in the calibration dataset matter. The authors also proposed a method to keep the quantization scale and zero point and change the integer code to achieve robustness to the distribution shift of input activation. The experiments shows in diferrennet PTQ methods and architectures the proposed method achieved better performance, and this method is a completely add-on post-hoc method to solve this discrepancy.
Critical strengths
This study is very impressive and very important for the PTQ community:
- Every researcher knows that reconstruction loss does not necessarily align with task loss. However, nobody deals with this problem as a central target of research. The authors ask it and provide a very large discrepancy between reconstruction loss and task loss.
- The proposed method is very easy to implement for existing PTQ.
- The authors conducted extensive experiments on several PTQ methods, and the experimental results are consistent.
Critical weaknesses
This paper is good, but readability is the problem.
- These authors clearly used an LLM heavily to write and organize this paper, as stated in the AI use statement. This is OK; however, some parts, especially the experimental settings in the appendix, are too dense for human readers. This is for LLM reading.
- The analysis of reconstruction loss and task loss is relatively weaker than the experiments of the proposed method.
- The proposed method is just a combination of existing methods, and the novely is the mainly focuses on the connection to PTQ.
Questions for authors
I hope the authors add a more comprehensive analysis of reconstruction loss and task loss. Different architectures have different calibration data, internal latent expression analysis, and activation analysis. Especially, I want to know whether the distribution shift really occurs.
5 comments
Response to Reviewer RHoh (Part 2/3)
2. Comprehensive Analysis of Reconstruction Loss and Task Loss (continued) [Weakness 2 & Question 1 (Part A)]
This continues the Llama-3.2-1B-Instruct AWQ/GPTQ W3A16 () single-layer analysis in Part 1.
Furthermore, when selecting from each layer’s candidate pool (Table R2), DRQ’s robust objective directly resolves this mismatch:
Table R2. Model NLL changes (nats/token) averaged over 112 single-layer interventions. Negative values mean lower (better) NLL. Each comparison reports its paired document-bootstrap 95% confidence interval.
| Scenario / text | Rec Initial | Robust Initial [95% CI] | Robust Rec [95% CI] |
|---|---|---|---|
| GPTQ / Calibration WT2 | [] | [] | |
| GPTQ / WT2 test | [] | [] | |
| GPTQ / C4 | [] | [] | |
| AWQ / Calibration WT2 | [] | [] | |
| AWQ / WT2 test | [] | [] | |
| AWQ / C4 | [] | [] |
Takeaway from Table R2: Selecting candidates by minimizing reconstruction loss worsens model NLL in five of six cases, whereas DRQ’s robust objective significantly lowers model NLL relative to both the initial model and reconstruction selection across every calibration and held-out dataset.
3. Algorithmic Novelty and Broad Applicability [Weakness 3]
Seamless compatibility with existing PTQ methods is a major strength rather than a weakness: it gives DRQ the flexibility to serve as a general, future-proof refinement framework as PTQ continues to evolve. Rather than replacing specific quantizers, DRQ resolves a fundamental objective mismatch shared across weight-only PTQ pipelines. Because it operates directly on the quantized integer grid, DRQ remains broadly applicable as new PTQ algorithms are developed. Beyond this plug-in flexibility, our novelty lies in two main aspects:
-
Theoretical contribution: Figure 1 and Figure 4 show that lower reconstruction loss can degrade model PPL even as calibration data grow. Theorem 3.1 derives the exact condition under which activation second-moment shifts reverse reconstruction rankings, and Theorem 3.2 reduces the worst-case objective to a tractable spectral penalty.
-
Algorithmic and practical contribution: Existing PTQ methods differ widely in their weight transformations, scaling, clipping, and calibration workflows. DRQ provides a unified discrete search that updates only integer codes on the fixed quantization grid while preserving scales, zero points, and inference kernels. Across our paper and new rebuttal experiments, DRQ improves eight diverse PTQ methods (GPTQ, AWQ, RTN, OmniQuant, GPTAQ, ParoQuant, TesseraQ, and SignRoundV2) across dense and MoE models with zero inference overhead.
Response to Reviewer RHoh (Part 3/3)
4. Does the Distribution Shift Really Occur in Practice? [Question 1 (Part B)]
Yes. Direct activation measurements across datasets show that shifting from WikiText-2 to C4 or Pile strongly amplifies activation energy along DRQ’s worst-case direction and sharply increases local reconstruction loss, whereas within-domain controls remain near zero.
Tables R3 and R4 measure activation shifts and their impact on layer reconstruction loss with fixed quantized weights on Llama-3.2-3B-Instruct (GPTQ seeds 0 and 1) and Llama-3.1-8B-Instruct (GPTQ seed 0). Relative to an independent WikiText-2 reference split (WT2-A), we compare two cross-domain datasets (C4 and Pile) against two within-domain controls (WT2-B and WT2-test), using 64 windows of 2,048 tokens per dataset. We also include a clean Full-Precision (FP) input control using the exact same GPTQ weight-error matrices to check whether activation shifts still occur without upstream quantization errors.
We report two metrics (median across blocks with 95% document-bootstrap confidence intervals over 2,000 draws):
-
Direction contrast: The activation energy ratio along DRQ’s worst-case direction minus the median ratio across 32 matched random directions (each relative to WT2-A).
-
Loss contrast (pp): The percentage-point change in local reconstruction loss on the target dataset relative to WT2-A, minus the corresponding change on the within-domain WT2-B control.
Table R3. Cross-domain activation shifts on C4 and Pile [with 95% bootstrap CI]. FP denotes full-precision layer inputs evaluated with the same GPTQ weight-error matrices.
| Input path / model | C4 direction contrast | C4 (pp) | Pile direction contrast | Pile (pp) |
|---|---|---|---|---|
| GPTQ / 3B seed 0 | [] | [] | [] | [] |
| GPTQ / 3B seed 1 | [] | [] | [] | [] |
| GPTQ / 8B seed 0 | [] | [] | [] | [] |
| FP / 3B seed 0 | [] | [] | [] | [] |
| FP / 3B seed 1 | [] | [] | [] | [] |
| FP / 8B seed 0 | [] | [] | [] | [] |
Table R4. Within-WikiText controls on WT2-B and WT2-test [with 95% bootstrap CI]. Definitions match Table R3. WT2-B has by construction.
| Input path / model | WT2-B direction contrast | WT2-test direction contrast | WT2-test (pp) |
|---|---|---|---|
| GPTQ / 3B seed 0 | [] | [] | [] |
| GPTQ / 3B seed 1 | [] | [] | [] |
| GPTQ / 8B seed 0 | [] | [] | [] |
| FP / 3B seed 0 | [] | [] | [] |
| FP / 3B seed 1 | [] | [] | [] |
| FP / 8B seed 0 | [] | [] | [] |
Takeaways from Tables R3 and R4:
-
Strong cross-domain directional amplification: Shifting from WikiText-2 to C4 or Pile amplifies activation energy along DRQ’s worst-case direction by to relative to matched random directions and increases local reconstruction loss by more than 50 percentage points beyond the within-domain control, with all 95% confidence intervals well above zero.
-
Near-zero within-domain fluctuation: Across independent WikiText-2 splits (Table R4), both directional and loss contrasts remain close to zero, confirming that the amplification in Table R3 reflects genuine cross-domain shift rather than sampling noise.
-
Intrinsic to input text rather than upstream quantization noise: Evaluating clean full-precision (FP) layer inputs with the same weight-error matrices produces the exact same directional amplification, proving that cross-domain text shifts alone excite these vulnerable directions even when all preceding layers are unquantized.
Response to Reviewer RHoh (Part 1/3)
Thank you for recognizing our work. We also believe that this study is very helpful to the PTQ community. We hope the analyses and experiments below resolve your concerns and encourage you to raise your score.
1. Readability of the Appendix [Weakness 1]
We have reorganized and simplified Appendices C to E to make them concise and easy to scan for readers.
In the revised manuscript, we consolidated all experimental settings into a unified summary table (Table 4), extracted inline MoE statistics into Table 11, removed dense inline timing lists in Appendix E.2, and added a numbered four-step workflow with an illustrative one-round example in Appendix C.
2. Comprehensive Analysis of Reconstruction Loss and Task Loss [Weakness 2 & Question 1 (Part A)]
Lowering calibration reconstruction loss can hurt model performance through two distinct mechanisms: an objective mismatch between local residuals and global task gradients on identical inputs, and overfitting to the sampled search positions.
As shown in Figure 1 of the submitted paper (summarized in Table F1), reducing a single layer’s reconstruction loss frequently increases whole-model PPL, even on the exact same calibration data:
Table F1. Summary of Figure 1 single-layer modifications.
To see why this happens even without distribution shift, consider the same fixed evaluation examples before and after a single-layer weight update. Let r0 denote the initial quantized layer’s output residual (W0x−Wx), and let δh be the layer-output change induced by the update. With input activations and all other layers fixed, the reconstruction-loss change and the end-to-end model-loss Taylor expansion are
\begin{aligned} \Delta L_{\mathrm{rec}} =\mathbb{E}\!\left[2\langle r_0,\delta h\rangle+\|\delta h\|^2\right], \qquad \Delta J =\mathbb{E}\!\left[\langle g_0,\delta h\rangle\right] +O\!\left(\mathbb{E}\|\delta h\|^2\right), \end{aligned}
where g0 is the gradient of the model task loss with respect to the layer output at the initial quantized model.
These two equations evaluate the exact same output change δh against different vectors (r0 versus g0). Whenever an update δh satisfies 2E⟨r0,δh⟩<−E∣δh∣2 while E⟨g0,δh⟩>0 dominates the higher-order remainder in ΔJ, it strictly lowers layer-output reconstruction error while simultaneously worsening model NLL on the exact same inputs. Theorems 3.1, 3.2, and B.2 complement this analysis by showing how activation shifts reverse reconstruction rankings and why simply adding more calibration data cannot eliminate an existing population-level objective mismatch.
We verified both mechanisms on Llama-3.2-1B-Instruct with AWQ and GPTQ W3A16 (g=128) across 560 single-layer updates (112 linear layers × 5 updates per layer), evaluating both losses on the exact same text (Table R1).
Table R1. Same-text reconstruction and NLL mismatch across 560 single-layer updates.
Takeaways from Table R1:
Same-text objective mismatch in AWQ: Over 60% of AWQ updates that reduce full-text calibration reconstruction loss simultaneously worsen model NLL on that exact same text, confirming the local-versus-global mismatch even without distribution shift.
Search-subset overfitting in GPTQ: All 560 GPTQ updates reduce reconstruction loss on the 512 sampled search positions, yet 100% of them increase reconstruction loss across the full calibration set, directly motivating DRQ’s checking-subset rules and directional penalty.
The candidate-selection comparison (Table R2) continues in Part 2.