- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how quantization distorts output token probabilities. To address the behavioral shift of quantized models relative to their full-precision counterparts, the authors propose a simple approach that can be applied directly post-quantization, without changing model weights, by adjusting two decoding settings per token: temperature and top-p sampling. The paper examines the extent to which these two changes can address the distributional shift.
Critical strengths
-
The authors propose a simple approach to align a quantized model’s output distribution with that of its full-precision counterpart by learning per-token temperatures and top-p thresholds, without updating the model’s weights. The authors investigate different divergence objectives for reducing this distributional shift and examine whether lower divergence consistently translates into better downstream task performance.
-
The authors provide a broad empirical evaluation: They assess distributional fidelity, downstream task performance, and generation behavior, including similarity to BF16 responses, diversity, repetition, and generated sequence length.
Critical weaknesses
-
W1: Although the forward-KL-trained controller reduces its optimized divergence, reverse KL, JSD, and total variation worsen relative to uncorrected quantization. These trade-offs limit the evidence that the proposed approach restores the full-precision model’s output distribution, beyond merely improving the training metric.
-
W2: Could the authors clarify which contributes more to fidelity to the full-precision model: temperature adjustment or top-p sampling? Currently, there is no top-p-only ablation, leaving its independent contribution unclear.
-
W3: The empirical evaluation is limited to post-trained Qwen3-4B with native MXFP4 quantization. A broader evaluation across model families and scales is important to establish whether the observed temperature and top-p effects generalize to different settings, particularly given previously reported differences in quantization performance and reliability across configurations in prior works [1,2].
[1] Dettmers, Tim, and Luke Zettlemoyer. "The case for 4-bit precision: k-bit inference scaling laws." International Conference on Machine Learning. PMLR, 2023.
[2] Ayadi, Sirine, et al. "Reliability Scaling Laws for Quantized Large Language Models." arXiv preprint arXiv:2607.10855 (2026).
Questions for authors
-
Q1: Could the authors explain why reducing the divergence does not consistently improve downstream performance or agreement with BF16-generated responses?
-
Q2: The authors aim to reduce the behavioral shift between quantized and full-precision models, yet Tables 1, 2, and 3 primarily report downstream performance and BLEU similarity to BF16 responses. To enable an overall assessment of the performance-fidelity trade-off, the main comparisons should present performance alongside distributional similarity and output-level agreement, on matched evaluation data.
-
Q3: While the authors look into different metrics, including token-level distributional fidelity, lexical similarity, diversity, repetition, and response length, I believe that using the MAUVE score [1] would be a useful measure of generation fidelity. The MAUVE score would assess how well the quantized outputs preserve the FP model's quality-diversity characteristics. Could you please report it for the different settings?
[1] Pillutla, Krishna, et al. "Mauve scores for generative models: Theory and practice." Journal of Machine Learning Research 24.356 (2023): 1-92.
9 comments
Response to Reviewer wh8J (2/2)
Q1: Why lower divergence does not always improve performance
We analyze this in the new Sections 4.7.1–4.7.2 [R.9.1–R.9.2]. Whether lowering a divergence improves performance and agreement with BF16 depends on whether that divergence targets the main source of error for the use case.
- Long reasoning under heavy degradation. Here the main source of error is looping: most of GPTQ3's reasoning responses repeat themselves until the length limit, while BF16's rarely do, so preserving BF16's behavior mostly means stopping the loops. Training on forward KL, which we find warms the model on average, breaks the loops and recovers both accuracy and similarity to BF16. Reverse KL also lowers its own divergence, but it cools the model and makes the repetition worse, so its accuracy drops (Table 2 [R2]).
- Short text. The quantized model does not loop, and its main error is spreading probability onto tokens BF16 would not pick, which reverse KL corrects, while forward KL makes summaries even more varied than BF16's (Table 3 [R21]).
- Reasoning under light degradation. With milder formats such as MXFP4, there is little accuracy to recover, but JSD, which weighs both kinds of error, brings which questions the quantized model answers correctly closer to BF16's (Figure 7 [R10]).
We therefore suggest TempQuant trained with forward KL when quantization costs much accuracy, with reverse KL for short text, and with JSD for reasoning under light degradation (Section 4.7.2 [R.9.2]). Each case shows its benefit:
| Setting | Per-step TempQuant | Task score | Behavior relative to BF16 |
|---|---|---|---|
| GPTQ3, reasoning, τ = 0.3 | Forward KL | MATH accuracy: 15.2% → 52.1% | At length limit: 97.6% → 44.4%; closeness and MAUVE also rise (see W1 and Q3) |
| GPTQ3, XSum summaries, τ = 0.6 | Reverse KL | ROUGE-2: 5.91 → 6.06 (BF16 6.52) | Unique summaries: 92% → 76% (BF16 75%) |
| MXFP4, MATH, τ = 0.6 | JSD | MATH accuracy: 92.7% → 93.3% | Answer disagreements: 106 → 91.5 |
In the second row, GPTQ3 makes 92% of its eight summaries per article unique against 75% for BF16, and reverse KL brings this to 76%, while summary quality also moves slightly toward BF16's. In the third row, accuracy barely changes, but JSD lowers the number of MATH attempts where MXFP4 and BF16 disagree on correctness from 106 to 91.5. Since two BF16 runs already disagree on about 67, this removes about a third of the disagreements that quantization adds, and the effect is significant across all four reasoning tasks and three temperatures (Holm p = 0.021; Table 33 [R30]).
Q2: Performance and fidelity together
We thank the reviewer for this suggestion. Table 2 [R2] now reports accuracy, the share of responses at the length limit, and closeness to BF16 on the same responses (see W1), and Table 3 [R21] does the same for summaries. Table 36 [R12] gives each policy's divergence, and Figure 7 [R10] and Tables 32 and 33 [R29 and R30] report correctness agreement with BF16.
Q3: MAUVE
We thank the reviewer for this suggestion, which further showed the benefit of our method. We added MAUVE for every reasoning task and temperature (Table 26 [R25]). At 0.3 and 0.6, both forms of TempQuant score above raw GPTQ3 on every task, for example at τ = 0.3:
| Task | BF16 vs BF16 | GPTQ3 | Global TempQuant | Per-step TempQuant |
|---|---|---|---|---|
| GSM8K | .951 | .458 | .662 | .652 |
| MATH | .939 | .579 | .879 | .883 |
| AIME | .958 | .548 | .841 | .830 |
| GPQA | .961 | .439 | .830 | .881 |
MAUVE compares whole generated responses and is not what TempQuant is trained on, so it is an independent check that TempQuant brings the quantized model's generations much closer to BF16's. At 0.3, per-step TempQuant closes about 85% of the gap between GPTQ3 and BF16's agreement with itself on both MATH (.579 → .883, against .939) and GPQA (.439 → .881, against .961).
We hope these revisions address the reviewer's concerns.
I thank the authors for their response and the additional results, including the MAUVE scores and top-p-only ablations.
The revised manuscript appears to differ substantially from the initial submission. It now presents global and per-step TempQuant as two main approaches, and the experiments section's focus has shifted from 4-bit MXFP4 to 3-bit GPTQ3. Several figures and tables also appear to have been added or substantially revised in the main body.
Could the authors provide a summary of all the changes made during the rebuttal period, covering both the methodology and the experimental setup?
Also, could the authors clarify how the closeness score is computed?
Response to Reviewer wh8J (1/2)
We thank the reviewer for the careful review and thoughtful suggestions. We especially appreciate the recommendations for the top-p-only ablation and MAUVE, which helped us better assess the contribution of each setting and how closely TempQuant matches BF16.
W1: Divergence trade-offs
We thank the reviewer for raising this concern. We agree that, depending on the training metric, the other divergences can rise, and Section 4.3 [R.3] now explains this trade-off. Knowledge distillation faces a similar trade-off: when a student cannot fully match its teacher, the choice of divergence determines which mismatches it prioritizes (Agarwal et al., 2024). TempQuant can be viewed as a restricted form of distillation, in which the full-precision model is the teacher, its quantized version is the student, and the student can lower the divergence only through its sampling settings, without changing its weights. Here, forward KL penalizes missing BF16's likely tokens, while reverse KL penalizes assigning probability where BF16 does not. Since temperature and top-p can only rescale and truncate the quantized model's own distribution, they cannot reorder its tokens, so the student cannot remove both kinds of mismatch at once: warming gives BF16's tokens back the probability that quantization took from them, but it also spreads probability onto tokens BF16 avoids, and cooling does the opposite. Lowering one divergence therefore tends to raise another. We also checked whether fixed temperature and top-p settings could avoid this trade-off, and no tested combination reduces all four divergences for GPTQ3 at τ = 0.3 or 0.6 (Appendix E.5 [R.10]).
However, even with this trade-off, our new data shows that both global TempQuant and per-step TempQuant (the per-token method from our submission) still restore the full-precision model's generations beyond the training metric (Table 2 [R2]):
Closeness summarizes how close the generated text is to BF16's in word overlap, variety, repetition, and responses hitting the length limit (Appendix E.7 [R.9.4]). MAUVE rises as well (see Q3).
W2: Temperature versus top-p
Following the reviewer's suggestion, we added a top-p-only ablation. Table 36 [R12] separates the contributions:
Both matter for divergence, with their relative contribution depending on temperature. For accuracy, adding temperature to the top-p-only global variation raises the four-task mean from 28.3% to 40.0% at 0.6 (Tables 2 and 24 [R2 and R35]). Adding a learned top-p to the per-step temperature-only controller lowers forward KL further at every temperature, with accuracy close to the temperature-only version (Appendix E.6 [R.11], Tables 23 and 24 [R34 and R35]).
W3: Generality
To address the reviewer's concern about generality, we added 3-bit Qwen3-8B, R1-Distill-Llama-8B, and Gemma 4 31B, as well as Qwen3-4B GPTQ3 and official AWQ (Table 12 [R17], Appendix E.10 [R.8.2]). At 0.6, global TempQuant raises Qwen3-8B MATH accuracy from 60.8% to 70.4% and Gemma's from 69.8% to 90.3%. R1-Distill-Llama-8B shows little change (67.3% to 67.8%), and we find that the largest accuracy gains occur in models that loop.