acceptodds
Under review as a conference paper at ICLR 2027

TempQuant: Learning Adaptive Temperatures For Quantized Language Models

Abstract

Large language models are increasingly deployed at low precision for applications ranging from reasoning to creative writing and beyond, yet quantization can degrade their performance. These applications often use different decoding settings, including sampling temperature and top-p, which control the randomness of token selection. We study the interaction between quantization and these settings by examining how quantization distorts output token probabilities. These distortions can change generated responses and affect task performance. We characterize the connection between quantization and temperature by measuring changes in generation-time entropy across deployed temperatures, and examine whether these changes can be understood as a temperature-like shift. We also investigate how much distributional error can be corrected through our intervention method, TempQuant. TempQuant is a compact learned predictor that adaptively sets the quantized model’s temperature and top-p value at each decoding step to minimize divergence between the full-precision and quantized probability distributions. At deployment, it uses only features of the quantized model’s out- puts, requiring no full-precision reference. We show that TempQuant can improve model accuracy and analyze how these corrections affect model behavior during generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wh8JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how quantization distorts output token probabilities. To address the behavioral shift of quantized models relative to their full-precision counterparts, the authors propose a simple approach that can be applied directly post-quantization, without changing model weights, by adjusting two decoding settings per token: temperature and top-p sampling. The paper examines the extent to which these two changes can address the distributional shift.

Critical strengths
  • The authors propose a simple approach to align a quantized model’s output distribution with that of its full-precision counterpart by learning per-token temperatures and top-p thresholds, without updating the model’s weights. The authors investigate different divergence objectives for reducing this distributional shift and examine whether lower divergence consistently translates into better downstream task performance.

  • The authors provide a broad empirical evaluation: They assess distributional fidelity, downstream task performance, and generation behavior, including similarity to BF16 responses, diversity, repetition, and generated sequence length.

Critical weaknesses
  • W1: Although the forward-KL-trained controller reduces its optimized divergence, reverse KL, JSD, and total variation worsen relative to uncorrected quantization. These trade-offs limit the evidence that the proposed approach restores the full-precision model’s output distribution, beyond merely improving the training metric.

  • W2: Could the authors clarify which contributes more to fidelity to the full-precision model: temperature adjustment or top-p sampling? Currently, there is no top-p-only ablation, leaving its independent contribution unclear.

  • W3: The empirical evaluation is limited to post-trained Qwen3-4B with native MXFP4 quantization. A broader evaluation across model families and scales is important to establish whether the observed temperature and top-p effects generalize to different settings, particularly given previously reported differences in quantization performance and reliability across configurations in prior works [1,2].

[1] Dettmers, Tim, and Luke Zettlemoyer. "The case for 4-bit precision: k-bit inference scaling laws." International Conference on Machine Learning. PMLR, 2023.

[2] Ayadi, Sirine, et al. "Reliability Scaling Laws for Quantized Large Language Models." arXiv preprint arXiv:2607.10855 (2026).

Questions for authors
  • Q1: Could the authors explain why reducing the divergence does not consistently improve downstream performance or agreement with BF16-generated responses?

  • Q2: The authors aim to reduce the behavioral shift between quantized and full-precision models, yet Tables 1, 2, and 3 primarily report downstream performance and BLEU similarity to BF16 responses. To enable an overall assessment of the performance-fidelity trade-off, the main comparisons should present performance alongside distributional similarity and output-level agreement, on matched evaluation data.

  • Q3: While the authors look into different metrics, including token-level distributional fidelity, lexical similarity, diversity, repetition, and response length, I believe that using the MAUVE score [1] would be a useful measure of generation fidelity. The MAUVE score would assess how well the quantized outputs preserve the FP model's quality-diversity characteristics. Could you please report it for the different settings?

[1] Pillutla, Krishna, et al. "Mauve scores for generative models: Theory and practice." Journal of Machine Learning Research 24.356 (2023): 1-92.

9 comments
AuthorsOpenReview ↗

Response to Reviewer wh8J (1/2)

We thank the reviewer for the careful review and thoughtful suggestions. We especially appreciate the recommendations for the top-p-only ablation and MAUVE, which helped us better assess the contribution of each setting and how closely TempQuant matches BF16.

W1: Divergence trade-offs

We thank the reviewer for raising this concern. We agree that, depending on the training metric, the other divergences can rise, and Section 4.3 [R.3] now explains this trade-off. Knowledge distillation faces a similar trade-off: when a student cannot fully match its teacher, the choice of divergence determines which mismatches it prioritizes (Agarwal et al., 2024). TempQuant can be viewed as a restricted form of distillation, in which the full-precision model is the teacher, its quantized version is the student, and the student can lower the divergence only through its sampling settings, without changing its weights. Here, forward KL penalizes missing BF16's likely tokens, while reverse KL penalizes assigning probability where BF16 does not. Since temperature and top-p can only rescale and truncate the quantized model's own distribution, they cannot reorder its tokens, so the student cannot remove both kinds of mismatch at once: warming gives BF16's tokens back the probability that quantization took from them, but it also spreads probability onto tokens BF16 avoids, and cooling does the opposite. Lowering one divergence therefore tends to raise another. We also checked whether fixed temperature and top-p settings could avoid this trade-off, and no tested combination reduces all four divergences for GPTQ3 at τ = 0.3 or 0.6 (Appendix E.5 [R.10]).

However, even with this trade-off, our new data shows that both global TempQuant and per-step TempQuant (the per-token method from our submission) still restore the full-precision model's generations beyond the training metric (Table 2 [R2]):

GPTQ3, τ = 0.3GPTQ3Global TempQuantPer-step TempQuant
Closeness to BF16 (BF16 = 100)43.771.071.2

Closeness summarizes how close the generated text is to BF16's in word overlap, variety, repetition, and responses hitting the length limit (Appendix E.7 [R.9.4]). MAUVE rises as well (see Q3).

W2: Temperature versus top-p

Following the reviewer's suggestion, we added a top-p-only ablation. Table 36 [R12] separates the contributions:

GPTQ3 policyForward KL, τ = 0.3τ = 0.6τ = 1.0
Uncorrected2.4091.4930.867
Global, top-p only1.0370.5480.364
Global, temperature only0.5640.5130.474
Global, temperature and top-p0.4510.3970.346

Both matter for divergence, with their relative contribution depending on temperature. For accuracy, adding temperature to the top-p-only global variation raises the four-task mean from 28.3% to 40.0% at 0.6 (Tables 2 and 24 [R2 and R35]). Adding a learned top-p to the per-step temperature-only controller lowers forward KL further at every temperature, with accuracy close to the temperature-only version (Appendix E.6 [R.11], Tables 23 and 24 [R34 and R35]).

W3: Generality

To address the reviewer's concern about generality, we added 3-bit Qwen3-8B, R1-Distill-Llama-8B, and Gemma 4 31B, as well as Qwen3-4B GPTQ3 and official AWQ (Table 12 [R17], Appendix E.10 [R.8.2]). At 0.6, global TempQuant raises Qwen3-8B MATH accuracy from 60.8% to 70.4% and Gemma's from 69.8% to 90.3%. R1-Distill-Llama-8B shows little change (67.3% to 67.8%), and we find that the largest accuracy gains occur in models that loop.

AuthorsOpenReview ↗

Response to Reviewer wh8J (2/2)

Q1: Why lower divergence does not always improve performance

We analyze this in the new Sections 4.7.1–4.7.2 [R.9.1–R.9.2]. Whether lowering a divergence improves performance and agreement with BF16 depends on whether that divergence targets the main source of error for the use case.

  • Long reasoning under heavy degradation. Here the main source of error is looping: most of GPTQ3's reasoning responses repeat themselves until the length limit, while BF16's rarely do, so preserving BF16's behavior mostly means stopping the loops. Training on forward KL, which we find warms the model on average, breaks the loops and recovers both accuracy and similarity to BF16. Reverse KL also lowers its own divergence, but it cools the model and makes the repetition worse, so its accuracy drops (Table 2 [R2]).
  • Short text. The quantized model does not loop, and its main error is spreading probability onto tokens BF16 would not pick, which reverse KL corrects, while forward KL makes summaries even more varied than BF16's (Table 3 [R21]).
  • Reasoning under light degradation. With milder formats such as MXFP4, there is little accuracy to recover, but JSD, which weighs both kinds of error, brings which questions the quantized model answers correctly closer to BF16's (Figure 7 [R10]).

We therefore suggest TempQuant trained with forward KL when quantization costs much accuracy, with reverse KL for short text, and with JSD for reasoning under light degradation (Section 4.7.2 [R.9.2]). Each case shows its benefit:

SettingPer-step TempQuantTask scoreBehavior relative to BF16
GPTQ3, reasoning, τ = 0.3Forward KLMATH accuracy: 15.2% → 52.1%At length limit: 97.6% → 44.4%; closeness and MAUVE also rise (see W1 and Q3)
GPTQ3, XSum summaries, τ = 0.6Reverse KLROUGE-2: 5.91 → 6.06 (BF16 6.52)Unique summaries: 92% → 76% (BF16 75%)
MXFP4, MATH, τ = 0.6JSDMATH accuracy: 92.7% → 93.3%Answer disagreements: 106 → 91.5

In the second row, GPTQ3 makes 92% of its eight summaries per article unique against 75% for BF16, and reverse KL brings this to 76%, while summary quality also moves slightly toward BF16's. In the third row, accuracy barely changes, but JSD lowers the number of MATH attempts where MXFP4 and BF16 disagree on correctness from 106 to 91.5. Since two BF16 runs already disagree on about 67, this removes about a third of the disagreements that quantization adds, and the effect is significant across all four reasoning tasks and three temperatures (Holm p = 0.021; Table 33 [R30]).

Q2: Performance and fidelity together

We thank the reviewer for this suggestion. Table 2 [R2] now reports accuracy, the share of responses at the length limit, and closeness to BF16 on the same responses (see W1), and Table 3 [R21] does the same for summaries. Table 36 [R12] gives each policy's divergence, and Figure 7 [R10] and Tables 32 and 33 [R29 and R30] report correctness agreement with BF16.

Q3: MAUVE

We thank the reviewer for this suggestion, which further showed the benefit of our method. We added MAUVE for every reasoning task and temperature (Table 26 [R25]). At 0.3 and 0.6, both forms of TempQuant score above raw GPTQ3 on every task, for example at τ = 0.3:

TaskBF16 vs BF16GPTQ3Global TempQuantPer-step TempQuant
GSM8K.951.458.662.652
MATH.939.579.879.883
AIME.958.548.841.830
GPQA.961.439.830.881

MAUVE compares whole generated responses and is not what TempQuant is trained on, so it is an independent check that TempQuant brings the quantized model's generations much closer to BF16's. At 0.3, per-step TempQuant closes about 85% of the gap between GPTQ3 and BF16's agreement with itself on both MATH (.579 → .883, against .939) and GPQA (.439 → .881, against .961).

We hope these revisions address the reviewer's concerns.

I thank the authors for their response and the additional results, including the MAUVE scores and top-p-only ablations.

The revised manuscript appears to differ substantially from the initial submission. It now presents global and per-step TempQuant as two main approaches, and the experiments section's focus has shifted from 4-bit MXFP4 to 3-bit GPTQ3. Several figures and tables also appear to have been added or substantially revised in the main body.

Could the authors provide a summary of all the changes made during the rebuttal period, covering both the methodology and the experimental setup?

Also, could the authors clarify how the closeness score is computed?

Reviewer_9raMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TempQuant learns per-token temperature and top-p adjustments to make quantized LLM outputs resemble a BF16 reference, using only quantized logits at inference. It substantially reduces its optimized divergence metric, but improvements in accuracy and generated-response fidelity are inconsistent.

Critical strengths
  1. Compact intervention: Only 165,048 controller parameters are trained; language-model weights stay frozen, and inference requires no BF16 teacher.
  2. Useful analysis: Temperature-only and joint oracles quantify the potential of decoding corrections while clearly acknowledging that they cannot repair token rankings.
  3. Careful reporting: The paper compares three divergence objectives, includes global and temperature-only controls, and reports uncertainty and unfavorable results.
Critical weaknesses
  1. Practical benefit remains unclear: Forward-KL training worsens total variation and XSum lexical similarity to BF16. Downstream gains vary, and the available matched controls do not establish an accuracy advantage for the full method (Tables 2–3, 14).
  2. Validation is limited: Evidence centers on Qwen3-4B with MXFP4; the Gemma pilot shows no accuracy gain. Reused development data, two training seeds, and unmeasured deployment overhead limit confidence in generalization and efficiency.
Questions for authors
  1. How sensitive are the gains and predicted controls to the KL uniform-mixture weight and smooth top-p approximation?
  2. On an untouched evaluation set, does adaptive temperature/top-p outperform tuned global settings across additional quantization formats, and at what latency cost?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 9raM (1/2)

We thank the reviewer for raising this. The new 3-bit data shows a clear practical benefit. On GPTQ3, both global TempQuant and per-step TempQuant (the per-token method from our submission) raise accuracy far above the uncorrected model, significant after Holm correction (Tables 2 and 10 [R2 and R15]):

GPTQ3, τ = 0.3, accuracy (%)GSM8KMATHGPQA
BF1694.395.140.3
GPTQ318.315.26.7
Global TempQuant81.753.025.0
Per-step TempQuant81.352.126.1

Global TempQuant, the variant of our method that learns a single temperature and top-p to minimize the divergence between the quantized and full-precision next-token distributions, can be applied today by changing the sampler settings, and it improves on the configuration Qwen recommends for its quantized releases (Table 1 [R1], τ = 0.6):

GPTQ3, accuracy (%)MATHGPQA
Default (0.6, .95)23.711.1
Qwen's recipe (presence penalty 1.5)33.817.7
Global TempQuant51.023.7

The XSum trade-off follows from the choice of divergence: forward KL warms the model on average, which breaks reasoning loops but makes summaries more varied than BF16's, while reverse KL keeps summaries closest to BF16 (Table 3 [R21]). We therefore suggest using TempQuant trained with forward KL when quantization costs much accuracy, TempQuant trained with reverse KL for short text, and TempQuant trained with JSD for reasoning under light degradation (Section 4.7.2 [R.9.2]).

W2: Validation

We agree that the original validation was limited. The Gemma pilot used MXFP4, which kept Gemma 4 31B within a point of BF16 on MATH (96.8% against 97.5%), so there was little accuracy for TempQuant to recover. At 3 bits, where Gemma loses far more, global TempQuant raises its MATH accuracy from 70% to 90% (Table 12 [R17]). We also added Qwen3-8B, R1-Distill-Llama-8B, and Qwen's AWQ release. On 64 fresh questions not used during development, with no refitting (Appendix D.1 [R.7.1]):

Forward KL to BF16, τ = 0.6, fresh questionsUncorrectedGlobal TempQuantPer-step TempQuant
GPTQ31.350.350.37
MXFP40.790.230.32

We now train five runs per loss, and the divergences vary by only a few percent (Table 21 [R32]). We also now measure the deployment overhead. Global TempQuant adds no cost, since it only changes the settings passed to the sampler, and our unoptimized per-step implementation adds 18–34% per token with the same peak memory:

Milliseconds per generated tokenBatch 1Batch 8Batch 64
Requested settings2.933.486.09
Global TempQuant2.893.415.89
Per-step TempQuant3.47 (+18%)4.64 (+33%)8.19 (+34%)

The predictor itself is small; most of this time goes to passes over the full vocabulary to compute its features and apply top-p, which our implementation does not optimize and which fused kernels could shorten (Appendix G [R.13]).

AuthorsOpenReview ↗

Response to Reviewer 9raM (2/2)

Q1: Sensitivity

We thank the reviewer for raising this concern. The results are not sensitive to these choices. Halving or doubling the KL smoothing weight or the smooth top-p sharpness changes held-out forward KL by at most 5% and the mean chosen temperature by at most 4% (Table 8 [R36]), and varying the smoothing weight from 1e-6 to 1e-3 leaves global TempQuant's forward KL unchanged (Table 14 [R6]).

Q2: Adaptive versus tuned global settings

We thank the reviewer for this question. We first compare global TempQuant with tuning the temperature on labeled answers. Simply choosing the setting that best matches the full-precision distribution finds nearly the same temperature as tuning, while needing no labels and far less compute: it uses the full-precision model's next-token probabilities on a few hundred unlabeled responses, while tuning GPTQ3 on 200 MATH questions took about 7,800 graded responses (Section 4.4 [R.5], Figure 6 [R8], Tables 17 and 18 [R9 and R10]).

We also evaluate what adaptive per-token settings add over a global setting. Per-step TempQuant is close to global TempQuant at 0.3 and 0.6 and more accurate at 1.0, significant after Holm correction (Table 10 [R15]):

GPTQ3, τ = 1.0Per-step minus global (points) [95% CI]
GSM8K+2.2 [+1.1, +3.3]
MATH+2.9 [+1.3, +4.5]

We would also like to clarify how we view the two forms. Rather than positioning the adaptive form as a competitor to the global one, our main contribution is the finding that much of the quantization error can be recovered through temperature, so quantized models should not be deployed at the same settings as their full-precision versions. This correction can be applied at a global or a per-token level, and we evaluate both.

While per-step TempQuant gains only modestly over the global setting here, the headroom for per-token settings is large, as our oracle experiment shows. Our analysis in Section 4.2 [R.2] explains why: different tokens need different corrections, with flipped tokens tending to need warming and preserved tokens cooling, which one setting cannot provide. An oracle that uses BF16 to choose the optimal setting at every token, the one that minimizes forward KL, recovers far more of the quantization error (Tables 36 and 37 [R12 and R13]):

GPTQ3, τ = 0.6Forward KL to BF16GSM8K accuracy (%)MATH accuracy (%)
GPTQ31.49347.925.0
Global TempQuant0.39784.453.1
Per-step TempQuant0.47079.756.3
Oracle per-step TempQuant0.24092.778.1
BF16096.693.4

Global TempQuant is already the best single setting on the training data, so the remaining headroom lies in better per-token predictions, which motivates future work on stronger per-step predictors.

We hope these revisions address the reviewer's concerns.

AuthorsOpenReview ↗

Summary of Changes Between Original and Revised Version

Dear Reviewer,

Since the revised version contains quite a few changes, we wanted to provide a summary to make it easier to track what changed between the original and revised versions. Throughout, "original" refers to the version first submitted for review, and "revised" to the version uploaded during the rebuttal.

Methodology. The core method remains the same as in the original version. Per-step TempQuant is still the main method: the per-token controller we originally proposed, with the same architecture, training objectives, and settings. Global TempQuant minimizes the same divergence to BF16, but with one temperature and top-p for each requested temperature, and already appeared in the original version as the fitted global control (original Section 4.3, Table 4), which we now fit with a finer search (revised Appendix E.3). Since the reviews raised the question of practical benefit, we now present it as a practical variation (revised Section 4.4), as it can be applied today by changing standard sampler settings.

Experimental setup. We have kept the MXFP4 results and moved them to revised Appendix E.9. Because MXFP4 loses at most 3.5 points from BF16 on GSM8K, MATH, and GPQA, it leaves little accuracy to recover, which made it harder to show the impact of the method. MXFP4 still shows strong behavior preservation, and we keep this result in the revised main text (revised Figure 7). 3-bit quantization was already part of our planned evaluation but was not ready at the time of the original submission. Since it degrades the model much more, it now leads the experiments (revised Sections 4.4–4.7). We also added Qwen3-8B, R1-Distill-Llama-8B, Gemma 4 31B, and Qwen's AWQ release (revised Table 12 and Appendix E.10).

Analysis. We expanded the analysis that motivates the method. We show that quantization makes the model run hotter (revised Section 4.1; original Section 4.2), that tokens whose top token flips under quantization need warming while tokens whose top token is preserved need cooling (revised Section 4.2, which measures on real tokens what original Figure 1 illustrated), and that the three divergences therefore select different temperatures (revised Section 4.3 and Appendix B). Following the reviewers' suggestions, we also added the top-p-only ablation, MAUVE, closeness, Holm-corrected tests, and runtime measurements.

Structure. Each experimental section of the original version continues in the revised version, with 3-bit results added and the MXFP4 numbers moved to revised Appendix E.9:

Original versionRevised versionChange
Section 4.1: Does correction preserve behavior and performance?Sections 4.6–4.7The GPTQ3 results come first, and the MXFP4 results from original Tables 1–3 are now in revised Tables 30 and 31.
Section 4.2: Does quantization change the temperature?Section 4.1The entropy analysis continues, and revised Figure 2 now shows GPTQ3 alongside MXFP4.
Section 4.3: How much error can TempQuant correct?Sections 4.4–4.5We make the same comparison of the fitted global setting, the learned controller, and the oracle, and the MXFP4 results from original Table 4 are now in revised Table 29.
Figure 1 and Appendix E.1: flipped and preserved tokens, and the three divergencesSections 4.2–4.3We expanded these illustrations into measurements on real tokens in the main text.

The controller diagram (original Figure 2) moved to revised Appendix C to make room.

Appendix. The Gemma MXFP4 pilot (original Appendix D.3) is replaced by Gemma 4 31B at 3 bits (revised Table 12). We removed the controller design experiments on inputs, history, on-policy updates, and training temperatures (original Appendices A.6 and B) to keep the appendix focused on the final method.

We hope this summary is helpful, and we would be glad to clarify any further questions.

AuthorsOpenReview ↗

Kind Reminder: Updated Manuscript

Dear Reviewer 9raM,

A quick note on the comment above: our summary of changes refers to our earlier revision. Please see our general comment, "Second Revised Version with Changes Marked in Blue", for the new revision. We hope this makes it easier to see what changed.

With the discussion period coming to a close, we would sincerely appreciate any further thoughts or questions you may have, and we would be happy to address them.

Thank you again for your time!

Reviewer_9C98OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript investigates per-token temperature scaling in quantised LLMs. In particular, the paper attempts to minimise per-token forward/backward KL divergence as well as JSD between the distributions of the base and quantised models, using a learned predictor based on quantised model logits and certain input features. The same is done to dynamically select top-p cutoffs.

The method shows mixed results on downstream tasks, yielding only a slight improvement over the baseline. Further, in an ablation study on GPQA, the paper reports "no accuracy benefit over the control" (Table 2, App D6). Finally, it is shown that the per-token KL divergence is partially reduced by the method used.

Critical strengths

The conceptual idea and implementation seem sound, and the experimental evaluation is well done. Similarly, the paper is mostly well-written, but these positive aspects are only mild.

Critical weaknesses

The method is not particularly interesting, and the experimental results show no improvement over the baseline. Consequently, it seems like there is no evidence that the proposed method works? I see no basis to accept this paper. If I overlooked some significant finding, I am happy to revise my score.

W1: Experimental Results cannot report significant improvements.

Appendix D6 and Table 2 present an ablation study with confidence intervals that include 0, i.e., they cannot show improvement. I suspect the same for the very mixed results in Table 1. This is not just my interpretation, but the manuscript itself states

Neither comparison establishes an accuracy benefit over its control.

W2: One-Parameter Per-Token KL Divergence to unquantised base model questionable Proxy for performance.

The paper reports successful reductions in KL but shows no significant improvement on downstream tasks. It seems the Metric is just not a good proxy for generation quality, or even generative similarity.

Questions for authors

Q1: Any significant improvements over the baseline?

Reading the paper, it seems like there is no benefit in any downstream task, at least none with established significance; is that correct? The same goes for lexicographic similarity. Technically, since three models are trained, i would also consider some form of multiple hypothesis testing fair.

Q2: Three Models are fit using three metrics; what is the motivation for each one?

Why use Backward/Forward KL or JSD over the other metrics? Is there a conceptual motivation?

11 comments
AuthorsOpenReview ↗

Response to Reviewer 9C98 (1/3)

We thank the reviewer for the careful review and for explaining the concerns about the experimental evidence. We appreciate the opportunity to address them with new results and statistical testing.

W1: Significance of improvements

We thank the reviewer for raising this concern. The ablation the reviewer quotes (submitted Table 2 and Appendix D.6) compared per-step TempQuant (the per-token method from our submission) on MXFP4 GPQA with two of our own controls, a fitted global temperature at 0.3 (+0.6 points, [−1.9, +3.1]) and a temperature-only controller at 0.6 (+0.5 points, [−1.3, +2.2]). Both controls are themselves temperature corrections, so these comparisons were between forms of our method rather than against the uncorrected model, and they were run on MXFP4, which loses at most 3.5 points from BF16 on GSM8K, MATH, and GPQA, leaving little room to show a benefit. The revised paper revisits both comparisons on GPTQ3. Across every task and temperature, per-step TempQuant is close to global TempQuant at 0.3 and 0.6 and significantly more accurate at 1.0 (Table 10 [R15]), and learning top-p on top of the temperature lowers divergence further with accuracy close to the temperature-only version (Appendix E.6 [R.11], Tables 23 and 24 [R34 and R35]).

Against the uncorrected model, the benefit is now clear. We have since added 3-bit GPTQ, where TempQuant raises mean reasoning accuracy from 10% to 40% at 0.3 and from 23% to 40% at 0.6 (Table 2 [R2]). Every gain below remains significant after Holm correction across all 75 comparisons (Table 10 [R15]), and the gains also hold when responses that hit the length limit are counted as wrong (Table 11 [R16]):

Global TempQuant minus GPTQ3 (points) [95% CI]τ = 0.3τ = 0.6
GSM8K+63.5 [+61.7, +65.2]+30.5 [+28.6, +32.3]
MATH+38.0 [+34.5, +41.6]+22.6 [+19.6, +25.5]
GPQA+18.3 [+14.2, +22.5]+14.2 [+10.3, +18.1]

The gains also carry over to other 3-bit models, and global TempQuant improves on the configuration Qwen recommends for its quantized releases, which adds a presence penalty to stop repetition (Tables 1 and 12 [R1 and R17], τ = 0.6):

Accuracy (%), τ = 0.6BaselineGlobal TempQuant
Qwen3-8B GPTQ3 (requested settings), MATH60.870.4
Gemma 4 31B GPTQ3 (requested settings), MATH69.890.3
Qwen3-4B GPTQ3 with Qwen's recipe (presence penalty), MATH33.851.0
Qwen3-4B GPTQ3 with Qwen's recipe (presence penalty), GPQA17.723.7

W2: KL as a proxy

We thank the reviewer for raising this. The submitted version covered only MXFP4, which leaves little accuracy to recover. To show the strength of KL as a proxy, we compare each policy's forward KL to BF16 with its generation quality and similarity on the new 3-bit results, where quantization damages the model. For generation quality, the lower a policy's forward KL to BF16, the higher its accuracy, and we show an oracle that minimizes forward KL at every token comes close to BF16 (Tables 36 and 37 [R12 and R13]):

GPTQ3, τ = 0.6Forward KL to BF16GSM8K accuracy (%)At length limit (%)
GPTQ31.49347.963.5
Per-step TempQuant0.47079.725.5
Global TempQuant0.39784.419.8
Oracle per-step TempQuant0.24092.73.1
BF16096.60.3

For similarity to BF16, the same ordering holds for closeness and MAUVE (Tables 2, 26, and 36 [R2, R25, and R12]). Closeness summarizes how close the generated text is to BF16's in word overlap, variety, repetition, and responses hitting the length limit, with BF16 at 100 (Appendix E.7 [R.9.4]):

GPTQ3, τ = 0.6Forward KL to BF16Closeness to BF16MAUVE, MATHMAUVE, GPQA
GPTQ31.49347.3.797.731
Per-step TempQuant0.47068.1.830.861
Global TempQuant0.39770.2.881.918

When quantization does little damage, similar accuracy can hide many answer flips, samples that the full-precision model answers correctly and the quantized model does not, or the reverse (Dutta et al., 2024). On MXFP4 MATH, the reverse KL and JSD controllers keep accuracy about the same while removing about a third of the extra answer flips with BF16 (Figure 7 [R10]).

AuthorsOpenReview ↗

Response to Reviewer 9C98 (2/3)

Q1: Significant improvements and multiple testing

At submission, most of our MXFP4 results (submitted Table 1) were not significant, since MXFP4 loses little accuracy on most tasks. The intervals that excluded zero were on the harder tasks, GPQA at 0.6 (+2.8 points, [0.5, 5.1]) and AIME at 0.3 and 0.6, and they were not yet corrected for multiple comparisons. The revised paper now has many significant gains. On 3-bit GPTQ, both forms of TempQuant significantly improve GSM8K, MATH, and GPQA at 0.3 and 0.6 (see W1), and on MXFP4 the forward KL controller's AIME gain at 0.3 remains significant (Table 30 [R27]).

For lexical similarity, the submitted XSum BLEU against BF16 (submitted Table 3) measured MXFP4 summaries, where every controller stayed within about 2.5 BLEU of native MXFP4. The revised paper measures similarity where quantization changes it more. On GPTQ3 reasoning, TempQuant raises word overlap with BF16's responses from 7.3 to 18.0 BLEU with global TempQuant and 16.6 with per-step TempQuant at 0.6 (Table 25 [R24]), and MAUVE rises with it (see W2). On summaries, the reverse KL controller brings GPTQ3's variety back to BF16's (Table 3 [R21]).

We thank the reviewer for the suggestion of multiple hypothesis testing and agree that it is appropriate. We now apply Holm correction across the comparisons for all three trained controllers, along with every task and temperature (Appendix D.4 [R.7.4]), and the gains reported above as significant remain so after it.

AuthorsOpenReview ↗

Response to Reviewer 9C98 (3/3)

Q2: Motivation for the three divergences

We thank the reviewer for this question, which the revised paper now addresses in Sections 4.2, 4.3, and 4.7 [R.2, R.3, and R.9].

Our training can be viewed as a restricted form of knowledge distillation: the full-precision model is the teacher, the quantized model is the student, and the student can only change its temperature and top-p. Since it cannot match the teacher exactly, it has to trade some errors against others, and as in on-policy distillation (Agarwal et al., 2024), the divergence decides which. We therefore train with all three and compare what each preserves (Section 4.3 [R.3]).

The three divergences select very different temperatures. At a requested 0.6, the best single temperature for GPTQ3 is 1.13 under forward KL, 0.73 under JSD, and 0.36 under reverse KL (Figure 4 [R5]), and per-step TempQuant trained with each loss picks a mean temperature of 1.40, 0.68, and 0.46 (Figure 5 [R6], Table 22 [R33]). On average, forward KL warms the model, reverse KL cools it, and JSD lands in between.

We trace this to the different corrections that different tokens need (Section 4.2 [R.2], Figure 3 [R4]). On most tokens, 83% for GPTQ3, quantization keeps BF16's most likely token and only spreads the probabilities out, so these tokens tend to be best matched by cooling: quantization warms the model on average (Section 4.1 [R.1]), and cooling undoes this. On the remaining 17% of tokens, however, the rankings flip and BF16's preferred token now ranks lower. Here, only a higher temperature gives it back probability, so these tokens tend to need warming. The flipped tokens are rare but each costs far more, so they carry about 80% of the error under every divergence we use (Figure 3 [R4]a).

The three divergences navigate this trade-off differently (Section 4.3 [R.3], Figure 8 [R7]). Forward KL is large when the quantized model gives little probability to a token BF16 considers likely, so a flipped token spikes forward KL, and forward KL prioritizes warming to correct the flipped tokens. Reverse KL is large when the quantized model puts probability where BF16 does not; warming spreads probability onto such tokens, so reverse KL tends to prefer cooling, even for the flipped tokens. JSD weighs both kinds of error and lands near the requested temperature. Comparing the three also addresses a question this trade-off raises: whether it is better to correct the flipped tokens, which are rare but carry most of the error, or the preserved tokens, which carry less of it but are far more common.

We find that the answer depends on the use case (Section 4.7 [R.9]). On long reasoning under heavy degradation, GPTQ3 collapses into repetition, and forward KL, which warms the most on average, breaks the loops and recovers the most accuracy. On short text, the quantized model's main error is spreading probability onto tokens BF16 would not pick, which reverse KL corrects by cooling on average. On reasoning under light degradation, JSD best preserves which questions are answered correctly. We therefore suggest TempQuant trained with forward KL when quantization costs much accuracy, with reverse KL for short text, and with JSD for reasoning under light degradation, as with milder formats such as MXFP4 (Section 4.7.2 [R.9.2]). Each controller's closeness to BF16 follows this split (Table 25 [R24]):

CaseExpectedClosest controller
GPTQ3 reasoning (heavy degradation, looping)Forward KLForward KL (68.1; JSD 50.9, reverse KL 41.7)
GPTQ3 summaries (short text)Reverse KLReverse KL (88.3; JSD 84.1, forward KL 78.8)
MXFP4 GPQA (reasoning, light degradation)JSDJSD (95.9; reverse KL 94.7, forward KL 92.7)

We hope these revisions address the reviewer's concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their careful reading of our paper and for their thoughtful feedback, and the Area Chair for overseeing the review process. At the time of submission, our results covered only MXFP4, which on Qwen3-4B loses at most 3.5 points from BF16 on GSM8K, MATH, and GPQA, leaving little accuracy to recover there. More aggressive 3-bit quantization was already part of our planned evaluation, but its results were not ready at the time of submission. We have since completed it, along with more models and a global form of TempQuant, which show the benefit of our method much more clearly, with statistically significant gains. The reviews helped us identify where the submission fell short, and we have uploaded a revised version.

A main concern across the reviews was practical benefit, the lack of established downstream gains, and whether TempQuant preserves the full-precision model's generations beyond the training metric. We have added more settings, especially more aggressive 3-bit quantization, and now show statistically significant gains on several reasoning tasks, along with closer similarity to the full-precision model's generations, measured directly with MAUVE, closeness to BF16, and agreement on which questions are answered correctly.

To make the method easier to use in practice, we added a global variation in the revision that uses the same objective to fit a fixed setting rather than per-token settings. It is deployable today by changing standard sampler settings, and we evaluate it alongside per-step TempQuant, the per-token method from our submission.

We would like to emphasize the broader practical argument supported by our results: quantized models should not automatically inherit the sampling settings recommended for their full-precision counterparts. Qwen, for example, recommends the same temperature and top-p for thinking-mode generation in its full-precision and quantized model cards. TempQuant shows that recalibrating these settings, at a global or per-token level, can reduce the distributional mismatch introduced by quantization, improve downstream performance, and bring the quantized model's generations closer to the full-precision model's.

The main additions are:

  • Broader evaluation: We added 3-bit Qwen3-4B, Qwen3-8B, R1-Distill-Llama-8B, and Gemma 4 31B, plus Qwen's official 4-bit AWQ release (Tables 2 and 12 [R2 and R17], Appendix E.10 [R.8.2]).
  • Analysis: To motivate the method, we added analysis of the interplay between temperature and quantization error: how quantization affects entropy, why tokens whose top token flips under quantization typically need warming while tokens whose top token is preserved need cooling, and how the three divergence objectives affect the temperatures chosen (Sections 4.1–4.3 [R.1–R.3]).
  • Stronger validation: Holm-corrected accuracy tests, MAUVE and correctness agreement, fresh evaluation questions, five-run divergence checks, smoothing sensitivity evaluation, and measured deployment overhead (Appendices D, E, and G [R.7–R.13]).

Reviewer concerns and our responses

  • Reviewer wh8J asked about improvements beyond the training metric, the roles of temperature and top-p, and applicability across models and quantization formats. We added MAUVE, correctness agreement, a top-p-only ablation, and broader evaluations, and clarified the divergence trade-offs.
  • Reviewer 9raM raised concerns about practical benefit, robustness, and deployment cost. We added broader comparisons, fresh-data divergence checks, repeated training, sensitivity checks, and latency measurements, and report where per-step adaptation helps over global settings.
  • Reviewer 9C98 raised questions about statistical significance, KL as a proxy for generation quality and similarity, and the motivation for the three divergences. We report significant 3-bit gains after multiple-comparison correction, show that lower KL tracks both accuracy and similarity to the full-precision model's generations, and added a section explaining why each divergence selects a different temperature and when each should be used.

We appreciate the opportunity to improve the paper through your feedback, and we thank the Area Chair and reviewers for their time and consideration.

AuthorsOpenReview ↗

Second Revised Version with Changes Marked in Blue

Dear Area Chair and Reviewers,

Since the changes in our earlier revision can be hard to see, we wanted to make it easier to see what was added. We have uploaded a new revision that keeps the original submission as it was and marks everything we added in blue. The new results are in Appendix R. Not every new section responds to a specific review comment, since many of these experiments, in particular the 3-bit quantization and the additional models, were part of our planned evaluation but were not ready at the time of submission. Our responses to each review point to the sections that address that reviewer's concerns.

Since we were not sure whether reviewers can see the history of all revisions, we have also included our earlier revision in the supplementary material, at tempquant-code/paper_versions/TempQuant_revised_reorganized.pdf. It is the more "camera-ready" version, which integrates the new results into the main text and appendix and reads more smoothly as a whole. Both versions contain the same data and experiments and differ only in formatting and writing. The new one simply makes it easier to see what changed from the initial submission.

We have also updated our previous responses so that they refer to both versions: each reference to the earlier revision is followed by the matching location in the current PDF in brackets, for example Table 2 [R2].

We thank the reviewers again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.