acceptodds
Under review as a conference paper at ICLR 2027

Temperature and Truncation Do Not Commute: Exact Conditions for Order-Independent Decoding

Abstract

Two systems that report identical sampling settings can sample from different distributions. Settings are reported as a short tuple such as temperature 0.7, top-p 0.95, while temperature scaling and truncation are applied in sequence to the logit vector and the order of the two steps is fixed by the inference implementation, not by the tuple. Temperature and min-p truncation are order independent under the reparameterization , a closed-form correction that depends on neither the model nor the vocabulary. Temperature and top-p are not order independent in general, and at fixed and fixed no change of the top-p threshold alone can make the two orders agree for every distribution, so the natural remedy of retuning is unavailable. Below unit temperature the effect is bounded by , and the same bound explains why it is largest at intermediate temperatures and vanishes as falls further. Over the complete vocabulary on approximately 290,000 next-token distributions per model, the corrected min-p pairing gives zero total variation distance in all 24 tested combinations, while the uncorrected min-p and top-p pairings reach 0.231 and 0.496 at and are zero only at . The Transformers library instantiates one of the two orders by default. On GSM8K, swapping the top-p order under a shared random stream changes exact-match accuracy by a statistically significant 1.34 percentage points. Reporting the composition order, or adopting the corrected min-p threshold, makes the reported configuration determine the sampled distribution.

Primary area:
generative models
Keywords:
language model decoding, stochastic decoding, temperature sampling, nucleus sampling, top-p sampling, min-p sampling, sampling order, reproducible evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZKBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the sampling distribution of LLMs changes under some operation over this distribution. Specifically, the authors focus on temperature scaling and truncation operation. They show that in most cases these operations do not commute. They then conduct some experiments to quantify how changing the orders of these operations impact the resulting distribution.

Critical strengths

The problem studied is clear and well defined. The overall writing is very clear and pleasant to read. The maths seems carefully conducted (though I did not check in details). Conclusions are clear.

Critical weaknesses

I have two main reserves for this paper. The first is that I am unsure that pointing out that truncation and scaling do not permute is a particularly strong contribution (though this is done cleanly here). The second is about the experiments.

  • authors used several but rather "less known" models. I wonder what are the justifications for using these models rather than others.
  • Then, I think the experiments overall support the claims of the paper. However, since most results are consequential from the above analysis, I could have appreciated other "empirical" analysis. For instance:
    • Do authors have insights about whether such discrepancies between distributions change with model size? Or is it rather linked to the model's internal design?
    • How does the discrepancies change with the task? For instance, on Figure 2 in appendix F, we see rather distinctive curves on GSM8k vs HumanEval. I wonder if authors have any insight about this.
    • A more refined analysis of the concrete implications could also be valuable. Table 2 from the main text already hints at this, but most of the paper is about "theoretical distributional measures" but more operational consequences, or model's sensitivity to this could be appreciated.

Overall, the paper supports its main claims. But since the main claim (temperature scaling and truncation do not commute) is not especially strong, I could have expected the paper to be more ambitious in exposing the practical implications of this. In the current form, it feels a bit shallow.

Questions for authors

See weaknesses.

5 comments

Thank you for the review and for finding the problem well defined and the writing clear. We followed each of your suggestions, and the resulting analyses are now the core of the revised results.

Choice of models. We chose the original three models from three developers to span next-token distribution shape (mean top-1 probability 0.43–0.61), the quantity the theory depends on. To separate size from design, the revision adds Qwen3 0.6B, 1.7B, and 4B, which are widely used and share one architecture family, training recipe, and tokenizer.

Model size or internal design? (§5.3, Fig. 1d–f). The shape of the next-token distribution decides. Within Qwen3 the top-pp TV falls with size, yet at a fixed top-1 probability all six models give nearly the same TV (0.0210–0.0220 for top-1 in [0.8, 0.9)). Reweighting each model to one top-1 mix removes most of the spread (τ=0.7\tau=0.7, p0=0.95p_0=0.95):

ModelPositions with top-1 ≥ p0p_0Observed TVReweighted TV
MiniCPM59%0.03570.0293
LFM2.59%0.03240.0285
Qwen3-0.6B15%0.03010.0279
Granite-4.123%0.02410.0263
Qwen3-1.7B26%0.02180.0254
Qwen3-4B29%0.01950.0246
Largest / smallest1.83×1.19×

Larger models place more predictions where the order cannot matter, since whenever the top token alone carries p0p_0 the two orders coincide exactly. A top-1 histogram from one forward pass therefore predicts the effect for a model and corpus.

Why do GSM8K and HumanEval differ? (§5.6, Fig. 3b). The answer is per-step entropy along the generation:

TaskBase entropy (nats/token)Per-step TVCoupled samples identical
HumanEval0.03–0.080.0005–0.00265–90%
GSM8K0.11–0.310.002–0.00913–67%
Open-ended1.4–2.80.02–0.10<3%

The ranking can be read from the untransformed model in advance.

Operational consequences.

  • Engines disagree by default (§5.5). llama.cpp applies temperature after top-pp and min-pp, while vLLM and Transformers apply it before, so the same setting samples different distributions locally and on a server. Min-pp settings convert exactly (α↦α1/τ\alpha\mapsto\alpha^{1/\tau}), and top-pp settings provably cannot (Prop. 4).
  • Generated text (Fig. 3d–f). At τ=0.7\tau=0.7, temperature-first raises degenerate continuations of Qwen3-1.7B from 13.5% to 19.9%, and at τ=1.2\tau=1.2 the top-pp pairing draws tokens 0.62–1.42 nats per token less likely. All repetition differences stay significant after Holm correction.
  • Benchmarks. On GSM8K, LFM2.5 shifts by 1.34 pp (CI [0.35, 2.31]), and the tighter order scores higher in 6 of 8 further settings below τ=1\tau=1, with the sign reversing for LFM2.5 at τ=1.2\tau=1.2, where Prop. 1 reverses the truncation direction. On HumanEval, the order trades pass@1 (+0.90 pp) against sample diversity.

Contribution. Non-commutation is the starting point of the paper, and its contribution is the exact boundary of order independence: a model- and vocabulary-free correction for min-pp, a proof that no threshold correction exists for top-pp, a bound within 1.4% of tight below unit temperature, and the condition under which the effect vanishes. The new results show what sets its size and where it changes generated text.

We hope the revision addresses your questions, and we would be glad to discuss any of them further.

Reviewer_ZKBDOpenReview ↗

Assessment confirmed

Dear authors, Thank you for your answer.

I also read the review of mrgG and it seems we largely agree on the proposed contribution.

The rebuttal does not clearly answer my concerns. First, I want to mention the rebuttal is much less clear than the paper, with lots of technical jargon and verbose conclusions that are very hard to process.

For instance:

  • span next-token distribution shape (mean top-1 probability 0.43–0.61), the quantity the theory depends on I find this wording really unclear and does not convincingly answer my question.
  • Model size or internal design? (§5.3, Fig. 1d–f). The shape of the next-token distribution decides. I do not understand this conclusion.
  • Reweighting each model to one top-1 mix removes most of the spread I do not understand what it means, nor am I sure you have enough empirical evidences to make such a claim.
  • Larger models place more predictions where the order cannot matter, since whenever the top token alone carries the two orders coincide exactly. Same issue.

Regarding the empirical insights, I do not clearly better understand the practical implications of the studied problem.

I decide to maintain my rating.

Thank you for reading our response of November 7 and for telling us directly where it fell short. To fit all our answers into one comment under the character limit, we compressed them until they depended on shorthand that we never defined. That was our mistake, since we could have used several comments, and we apologize for the time it cost you. We answer again in two comments and begin with the sentences you quoted.

Short answers.

  1. Model size or model design? Within the Qwen3 family the effect falls by 35% from 0.6B to 4B parameters. Most of that fall comes from one measurable property, how often the model is confident about the next token.
  2. Why these models? The first three differ in that property and come from three developers. The revision adds Qwen3 at three sizes to separate size from design.
  3. Why do GSM8K and HumanEval differ? The model is uncertain at more steps in math than in code.
  4. What should a user do? Report the order together with the settings, convert min-pp thresholds with one formula, and for top-pp set the same order in the engine, because no change of the top-pp value makes the two orders agree.

The setting, in one example. With next-token probabilities 0.6, 0.3, and 0.1, temperature 0.5, and top-pp 0.7, applying top-pp first leaves two tokens and samples the second one 20% of the time, while applying temperature first leaves one token and never samples the second. Reported settings do not say which order was used. Min-pp is a second common rule, which keeps the tokens whose probability is at least a set fraction of the top token's probability.

What we measure. For each next-token prediction we compute the total variation distance between the sampling distributions of the two orders. A distance of 0.03 means that the two orders must choose different tokens in at least 3 of every 100 draws. The numbers below use temperature 0.7 and top-pp 0.95 on the WikiText-103 test set.

"The shape of the next-token distribution decides." By shape we meant how much probability the model gives to its top candidate, which we call its confidence at that position. We grouped all positions by this probability. Each cell is the average distance between the two orders.

Probability of the top candidateMiniCPM5LFM2.5Qwen3-0.6BGranite-4.1Qwen3-1.7BQwen3-4B
below 20%0.04680.04640.04520.04400.04290.0417
20 to 40%0.04370.04210.04140.03970.03810.0369
40 to 60%0.03710.03520.03450.03090.02960.0280
60 to 80%0.02910.02850.02710.02410.02320.0222
80 to 90%0.02100.02200.02150.02150.02140.0213
90 to 95%0.00830.00890.01010.01230.01250.0130
95% or more000000

Down each column the distance falls from about 0.045 to zero. Across each row the six models, from four developers and from 0.6 to 4 billion parameters, differ by at most 0.009. Confidence at a position tells much more about the order effect than the identity of the model does. What remains within a row is small and orderly, with the larger Qwen3 models slightly lower in the first four rows.

"Predictions where the order cannot matter." The last row is zero by a short argument and needs no data. If the top token has probability 97% and top-pp is 0.95, then at any temperature of 1 or below top-pp keeps only that token whichever operation comes first. Models differ in how often they are in this row. It holds 9% of positions for MiniCPM5 and LFM2.5, 15% for Qwen3-0.6B, 23% for Granite-4.1, 26% for Qwen3-1.7B, and 29% for Qwen3-4B. Within Qwen3 the share grows with size.

"Reweighting each model to one top-1 mix." This was shorthand for a standard adjustment, the one used when death rates of two countries are compared after adjusting for age structure. We recomputed the average of each model as if all six had the same share of positions in each row. The ratio between the largest and the smallest model average falls from 1.83 to 1.19, and within Qwen3 the fall from 0.6B to 4B shrinks from 35% to 12%. Most of the difference between models therefore comes from how often each model is confident. Min-pp with one threshold in both orders shows the same pattern (2.02 to 1.14, Section 5.3).

What the evidence covers. The table covers 1.77 million predictions, which is every position of the test set for six models from four developers. The last row is proved for any model. The other rows are measured at temperature 0.7 and top-pp 0.95 on models of up to 4 billion parameters, with the size trend read from the three Qwen3 models, which share one design. The measured claim is that confidence explains most of the difference between these models, and the theory holds for any model of any size.

Reviewer_RfNBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a seeminlgy minor but consequential point about the order in which temperature and truncation is applied during sampling. For instance, for the top-p sampling algorithm most papers would report a combination of temperature and top_p value, without specifying the order in which they were applied. The two orders don't necessarily commute for all kinds of sampling algorithms. The paper shows that top-k commutes with temperature, but top-p and min-p do not commute by default. They further show that min-p could be made commutative by replacing α\alpha with ατ\alpha^\tau. The logical argument is easy to follow and is explained clearly in the paper. Empirically the paper (1) Measures the total variation distances over 870K next token distributions (2) Measures the impact on a downstream metric such as accuracy on GSM8K Math benchmark.

Critical strengths
  • S1: The paper identifies an important but overlooked point in LLM generation evaluations: reporting the numerical values for sampling parameters alone is not sufficient for specifying the final next token sampling distribution unless the order is also known.
  • S2: The condition for threshold correction for min-p is exact, simple and directly usable.
  • S3: The paired generation experiment was carefully designed which uses same random draws for both orders and shows at what token position do the two sequences first differ and how it propagates to downstream benchmark accuracy.
Critical weaknesses
  • W1: In the abstract and Section 5, the authors claim that the top-p order effect vanishes as temperature goes to 0. This is not necessarily true. In the case, when the maximum probability token is tied. For e.g. consider the next token distribution p=(0.3,0.3,0.2,0.2)p=(0.3, 0.3, 0.2, 0.2) and p0=0.4p_0=0.4. Applying the top_p first and then temperature gives (1/2,1/2,0,0)(1/2,1/2,0,0) whereas temperature first and then top_p gives (1,0,0,0)(1,0,0,0). The TV between them remains non zero. This statements needs to be appropriately qualified with the caveat. Even with finite precision floating point values, this is a possibility. These issues do not affect Prop. 1 to 4, but they weaken the claimed account of all four rules.

  • W2: The experimental section is thin. The evidence for downstream impact is limited to just a 1.2B model and on a couple of benchmark GSM8K and HumanEval with one decoding configuration. The paper falls short of studying how the downstream effect varies across models, tasks and sampling parameters. The paper also does not report pass@k for HumanEval

  • W3: While somewhat ambiguously reported, IMHO the observation that temperature and the truncation parameter do not commute is fairly well known in the community. One concrete example is the text watermarking community that has handled this question at the outset to avoid potential confusion. Perhaps it is not as well documented elsewhere. My overall concern here is about the significant contribution that this paper claims to make.

Questions for authors
  1. Can the authors provide a clarification on the issue raised in Weakness 1 above about handling the case of ties? How do the results change accounting for this?

  2. The paper says that min-k is unchanged by temperature. Does this hold for the full algorithm from Ding et al. What are some situations in which this commutativity is desirable other than avoiding reporting ambiguity?

  3. Many truncation samplers were initially developed to avoid the problem of degenerate text or repetitions etc. which might math and code generation benchmarks might not isolate effectively? Why is the evaluation limited to these?

5 comments

Thank you for the detailed review. We appreciate your note that the min-pp correction is exact, simple, and directly usable, and that the paired design is careful. Every change below is in the revised PDF.

W1/Q1. Ties at the top. Thank you for this precise observation. The revision makes the condition explicit as Corollary 1 (§4.1, proof in App. A.4). With kk tokens tied for the largest probability, TV tends to 0 as τ→0\tau\to0 whenever k≤1/(1−p0)k \le 1/(1-p_0). A positive limit therefore needs more than 10 exactly tied tokens at p0=0.9p_0=0.9 and more than 20 at p0=0.95p_0=0.95, and with two tied tokens it needs p0≤1/2p_0 \le 1/2. In floating point the boundary case k=1/(1−p0)k=1/(1-p_0) can also give 1/k1/k once the tail mass underflows, and our measurement counts it as positive. As you note, Props. 1–4 are unaffected, and Prop. 2 still bounds the limit by 1−p01-p_0. Measured on all 1.77M positions (Table 7):

Quantity (six models)Value
Positions with a tie for the top bfloat16 logit2.5–4.0%
Mean number of tied tokens2.03–2.11
Positions with a positive τ→0\tau\to0 limit (p0p_0 = 0.9, 0.95)<0.005%
TV at τ=0.1\tau=0.1, tied vs. untied positions0.007–0.011 vs. 0.002–0.005

W2. Breadth, and pass@k. The study now covers six models, including a Qwen3 size ladder, and three tasks, with all 25 comparisons corrected jointly (Holm). HumanEval pass@k (τ=0.7\tau=0.7; difference = temperature first minus truncation first, pp; greedy pass@1 is 56.7% for LFM2.5 and 70.1% for Qwen3-1.7B):

PairingΔpass@1Δpass@16Δ distinct programs
LFM2.5, top-pp+0.90 [0.23, 1.60]−0.20−0.41
LFM2.5, min-pp uncorrected+0.66−1.42−0.80
Qwen3-1.7B, top-pp+0.05−1.02−0.40

The tighter temperature-first order wins at pass@1, and truncation-first keeps more tokens, produces more distinct programs, and has the higher pass@16 estimates, the precision-diversity trade that Prop. 1 predicts. On GSM8K (four models, Table 8), the tighter order scores higher in 6 of 8 settings below τ=1\tau=1, and in all 8 when answers given outside the required format are also read.

W3. Novelty. Those who build sampling chains treat the order as an engineering choice, as llama.cpp's configurable chain and watermarking schemes show. The paper establishes what that choice implies: (1) which rules can be reconciled, and by which exact map (min-pp, α↦ατ\alpha\mapsto\alpha^\tau, free of model and vocabulary); (2) that no threshold change reconciles top-pp (Prop. 4); (3) a bound of 1−p01-p_0 below unit temperature, within 1.4% of tight on real distributions; (4) when the effect vanishes; (5) its measured size. These results matter for current software, since llama.cpp and vLLM default to opposite orders (§5.5).

Q2. Min-kk. The cliff index of Ding et al. depends only on logit gaps divided by the logit range RR, so dividing the logits by τ\tau leaves it unchanged, ties included. The fallback size ⌊c/R⌋\lfloor c/R\rfloor (c=3c=3 in their paper) becomes ⌊cτ/R⌋\lfloor c\tau/R\rfloor when temperature comes first, so the full rule commutes with temperature whenever the fallback is inactive in both orders, in particular when R>cmax⁡(1,τ)R>c\max(1,\tau) (App. D). Beyond reporting, commutativity lets (a) settings move between engines with different orders, (b) any system that recomputes sampled-token probabilities, such as rollout-trainer importance ratios in RL, match the sampler, and (c) an engine move temperature within its chain, for example into a fused kernel.

Q3. Degeneration. We added open-ended continuation of 300 WikiText-103 paragraphs (Fig. 3d–f, Table 10). For both rules, temperature-first keeps fewer tokens below τ=1\tau=1 and more above it, and the measures move accordingly in all six settings:

SettingMeasureTemp. firstTrunc. first
Qwen3-1.7B, top-pp, τ=0.7\tau=0.7degenerate continuations19.9%13.5%
Qwen3-1.7B, top-pp, τ=0.7\tau=0.7distinct 3-grams58.0%62.8%
Qwen3-1.7B, min-pp, τ=1.2\tau=1.2degenerate continuations5.2%9.6%
LFM2.5, top-pp, τ=1.2\tau=1.2log-prob per token−4.96−3.54
Corrected min-pp (both models)all measuresidenticalidentical

All repetition differences stay significant after Holm correction.

We hope these changes address your concerns, and we would welcome any further questions.

This note continues our response of November 7, about ten days ago. Thank you again for the detailed review. Your remark on ties became a new result in the paper. We have not yet heard whether the revision settles your points, and we would value your view. Our first response was compressed, and here we spell out the parts it left implicit.

W1 and Q1. Ties. Your observation is correct, and it made the paper more precise. The abstract and Section 5.1 now carry the qualification, and the revision states the condition as Corollary 1 (Section 4.1). With kk tokens tied for the largest probability, the total variation between the two orders can have a positive limit as the temperature goes to zero only if k>1/(1−p0)k > 1/(1-p_0), which means more than 10 tied tokens at p0=0.9p_0 = 0.9 and more than 20 at p0=0.95p_0 = 0.95. For two tied tokens the condition is p0<0.5p_0 < 0.5. Your example, (0.4, 0.4, 0.2) with p0=0.5p_0 = 0.5, lies on the boundary. There the distance is zero at every positive temperature in exact arithmetic, and it becomes 0.5 once the tail probability underflows in floating point, which is the finite-precision case you raised. Our measurement counts this boundary case as positive.

On the 1.77 million measured positions (Table 7), ties for the largest 16-bit logit occur at 2.5 to 4.0% of positions and involve 2.03 to 2.11 tokens on average, and the limit is positive at fewer than 0.005% of positions in every model. At temperature 0.1 and p0=0.95p_0 = 0.95 the mean distance is 0.007 to 0.011 at tied positions and 0.002 to 0.005 elsewhere, and at tied positions it roughly halves between temperature 0.25 and 0.1. Propositions 1 to 4 are unaffected, as you noted.

W2. Breadth of the experiments.

Submitted versionRevised version
Models36, adding Qwen3 at 0.6B, 1.7B, and 4B
Next-token distributions870 thousand1.77 million
Temperatures47, from 0.1 to 1.5
Tasks with generated textGSM8K, HumanEvalGSM8K, HumanEval, open-ended continuation
GSM8K accuracyLFM2.5 at one setting4 models, with temperatures 0.5 and 1.2 and p0=0.9p_0 = 0.9 added
HumanEvaldistinct programs per problempass@1, 4, 8, and 16 for LFM2.5 and Qwen3-1.7B
MultiplicitynoneHolm adjustment over the 25 added comparisons

HumanEval pass@kk (Table 9). With 16 samples per problem at temperature 0.7, the order that keeps fewer tokens, temperature first, gives the higher pass@1 in all three comparisons. For LFM2.5 with top-pp the gain is 0.90 points, with a 95% interval from 0.23 to 1.60 before multiplicity adjustment. Truncation first gives 0.4 to 0.8 more distinct programs per problem and the higher pass@16 point estimate in all three comparisons, by 0.20 to 1.42 points. The order trades the precision of a single sample against the variety of many.

GSM8K (Table 8). The 1.34-point shift for LFM2.5 (interval 0.35 to 2.31) is joined by eight comparisons below temperature 1 on four models. The order that keeps fewer tokens scores higher in 6 of the 8, and in all 8 when answers outside the required format are also read. The added shifts reach 0.90 points. Taken one at a time their intervals include zero, and what they contribute is a direction that follows the theory.

W3. Is the observation already known? We agree that people who build samplers know the order can matter, and that watermarking work fixes the point where its modification enters the chain. The min-pp identity is short, which is part of its value, as your S2 notes. Our claim is that the consequences around it had not been worked out and are not all obvious.

  • Min-pp has a closed-form conversion that involves neither the model nor the vocabulary.
  • For top-pp, no replacement threshold makes the two orders agree for every distribution (Proposition 4). Two rules that look alike fall on opposite sides.
  • At temperatures of 1 or below the top-pp distance is at most 1−p01-p_0, and single positions reach 0.0493 at p0=0.95p_0 = 0.95.
  • The distance vanishes as the temperature goes to zero except under the tie condition of Corollary 1.
  • Its size on real models follows how often they are confident, and its consequences are largest in open-ended text.

Fixing one order, as watermarking work does, removes the ambiguity inside one system. It does not tell two systems with different orders how to agree, and llama.cpp and vLLM use opposite orders. For min-pp the answer is one line, since a threshold of 0.1 applied before temperature 0.7 equals 0.037 applied after it. For top-pp the answer is that no threshold change works and that the order itself must be matched or reported. To our knowledge neither statement appears in earlier work, including the watermarking literature.

Q2. Min-kk. The cutoff of min-kk (Ding et al.) is the rank that maximizes a weighted gap between neighboring sorted logits, where each gap is divided by the logit range and by the rank. Dividing all logits by the temperature divides every gap and the range by the same factor, and the cutoff is unchanged, ties included. The full algorithm also has a fallback size computed from the logit range, which does change with temperature. The full algorithm is therefore order independent whenever the fallback does not exceed the cutoff in either order, for example when the logit range exceeds 3max⁡(1,τ)3\max(1,\tau) (Appendix D).

Order independence helps in three situations beyond reporting. Settings can move between engines that apply different orders. A program that recomputes the probability of a sampled token, as reinforcement learning from sampled outputs does for importance ratios, obtains the right value without copying the internal order of the sampler. An engine can also move the temperature step inside its chain without changing what it samples.

Q3. Degenerate and repetitive text. We agree that this is the setting truncation sampling was designed for, and the revision adds it (Section 5.6, Table 10). LFM2.5 and Qwen3-1.7B continue the openings of 300 WikiText-103 paragraphs, four times each, for up to 256 tokens. The table gives the fraction of repeated 4-grams within a continuation.

Model, rule, temperatureTemperature firstTruncation first
Qwen3-1.7B, top-pp 0.95, 0.731.73%26.95%
Qwen3-1.7B, top-pp 0.95, 1.29.24%11.78%
Qwen3-1.7B, min-pp 0.1, 1.219.50%24.07%
LFM2.5, top-pp 0.95, 0.75.78%4.24%
LFM2.5, top-pp 0.95, 1.20.23%0.50%
LFM2.5, min-pp 0.1, 1.22.76%4.13%
Both models, min-pp with the ατ\alpha^\tau correctionidenticalidentical

All six differences stay significant after the Holm adjustment over the 25 added comparisons. Below temperature 1 the temperature-first order keeps fewer tokens and repeats more, and above temperature 1 it keeps more tokens and repeats less, which is the direction the theory gives for both rules. The share of degenerate continuations, in which more than half of the 4-grams repeat, is 19.9% against 13.5% for Qwen3-1.7B at temperature 0.7 (difference 6.4 points, interval 3.9 to 8.9), and for LFM2.5, whose continuations rarely degenerate (0.8% against 0.4%), the difference shows in the repeated 4-grams of the table. Above temperature 1, temperature first also draws tokens that the model rates as less likely. For LFM2.5 with top-pp at temperature 1.2 the mean log-probability per token is −4.96 against −3.54.

We hope this resolves the concerns you raised. If one of them remains open, we will gladly continue the discussion.

Reviewer_mrgGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The central observation of the paper is that if one performs temperature sampling and a truncation to a model next token probability, one may need to decide which of the operations comes first. While the choice of top-k tokens in top-k sampling is unaffected by the temperature map, other decoding strategies such as top-p are heavily affected by temperature sampling. This effect has been observed before (the paper of Ding et al which the authors cite, and in the fact that Llama.cpp lets you decide which order to perform temperature sampling and top p) but to the best of my knowledge has not really been studied before.

The authors study the size of the effect for different decoding strategies, both experimentally and with theoretical bounds for top-p. For many other decoding strategies they show that there is no effect in swapping the order of truncation and temperature sampling.

Critical strengths

The authors do a good job of experimentally checking the size of the effect of swapping the order of temperature and top-p sampling. The proofs are clear and easy to follow.

The effect that the authors write about is not particularly well known, and deserves to be better known among those doing theoretical research on language models. The title is excellent in this regard, neatly saying something that more theoretical ML researchers ought to know.

Critical weaknesses

There is no critical weakness in the document. This is a competently done study and I fully agree with the authors conclusion section. But beyond the headline statement which I think should be better known, I don't find much really valuable in the document. It is useful to know that temperature and top-p do not commute and to be careful of this when performing experiments in NLP, but I didn't really feel I got much more out of reading the full eight pages.

I was interested to see that reversing the order of temperature and truncation could affect score on GSM8K by as much as 1.3%, I would have guessed lower. But this was for a single model and the 95% confidence interval comes out at a much more modest 0.35%.

Questions for authors

I don't have any technical questions or clarification requests. I think the study is competently done. I would be very likely to accept the work if I were reviewing for an NLP focussed workshop.

4 comments

Thank you for the careful review. We are glad you found the study competently done, the proofs clear, and the title a statement that theoretical researchers ought to know. You asked what the paper offers beyond its central observation. The revision adds four results that go beyond it.

1. A predictive law for the size of the effect (§5.3, Fig. 1d–f). The effect is set by the shape of the next-token distribution, not by the model. At a fixed top-1 probability, six models from four developers (0.6B–4B) give nearly the same TV, for example 0.0210–0.0220 for top-1 in [0.8, 0.9), and reweighting them to one top-1 mix shrinks the spread across models from 1.83× to 1.19×. Wherever the top token alone carries p0p_0, the orders coincide exactly for every τ≤1\tau\le1. One forward pass therefore predicts the effect for a new model or corpus before either order is run.

2. Both orders are in active use (§5.5). Widely used engines place temperature on opposite sides of the truncation:

EngineStandard sampler order
llama.cpptop-kk → top-pp → min-pp → temperature
vLLMtemperature → min-pp → top-kk/top-pp
Transformerstemperature → top-pp → min-pp

The same "τ=0.7\tau=0.7, top-pp 0.95" thus samples different distributions in a local and a server deployment, with TV of 0.020–0.036, rising to 0.21–0.42 at τ=1.5\tau=1.5. Prop. 3 gives the exact min-pp conversion between engines, α↦α1/τ\alpha\mapsto\alpha^{1/\tau}, and Prop. 4 proves that no conversion exists for top-pp.

3. Large effects where truncation matters most (§5.6, Fig. 3d–f). In open-ended continuation, temperature-first at τ=0.7\tau=0.7 raises the share of degenerate continuations of Qwen3-1.7B from 13.5% to 19.9%. At τ=1.2\tau=1.2 the direction reverses, as Prop. 1 predicts, and under the top-pp pairing sampled tokens become less likely under the model by 0.62–1.42 nats per token. All repetition differences stay significant after Holm correction across the 25 comparisons.

4. A precise vanishing condition (Corollary 1). The low-temperature limit is zero unless more than 1/(1−p0)1/(1-p_0) tokens tie exactly for the largest probability, which happens at fewer than 0.005% of positions.

GSM8K across models. The 1.34 pp shift carries a paired hierarchical bootstrap interval over seeds and problems that excludes zero. To test whether its direction is systematic, we added eight settings below unit temperature: LFM2.5 at τ=0.5\tau=0.5 and at p0=0.9p_0=0.9, and Qwen3 0.6B, 1.7B, and 4B under both pairings (Table 8). By Prop. 1, temperature-first is the tighter truncation in these settings, and it scores higher in 6 of 8, and in all 8 when answers given outside the required format are also read. Above unit temperature Prop. 1 reverses the roles, and the sign for LFM2.5 reverses accordingly (−0.66 pp). The size of the shift follows the per-step entropy of the task. GSM8K traces have base entropy of only 0.11–0.31 nats per token, so the per-step TV is 0.002–0.009, and although 33–87% of coupled completions change, most reach the same answer. Where entropy is higher, the same mechanism produces the 6.4 pp change in degeneration above.

Because the result concerns how a sampling distribution is specified, it applies wherever samples are drawn, including pass@kk and self-consistency evaluation, deployment across engines, and systems that recompute the probabilities of sampled tokens. We would be glad to clarify anything further.

This note follows our response of November 7, about ten days ago. Thank you again for the careful review. We have not yet had the benefit of your view on the revision, and we would value it while the discussion is open. Our first response was compressed, and this one is organized around the two remarks in your review.

Beyond the title. Your review summarized the theory accurately, and we do not repeat it. You found little beyond the headline statement in the version you read. The revision extends that version in three ways.

1. What explains the differences between models (Section 5.3). The submitted version showed that the effect concentrates where the next-token distribution is uncertain. The revision shows that this accounts for most of the difference between models and between model sizes. We grouped 1.77 million next-token predictions of six models by the probability of the top token. At temperature 0.7 and top-pp 0.95, the mean total variation between the two orders falls from about 0.045 where that probability is below 20% to zero where it reaches the top-pp value, and the zero is proved for every temperature of 1 or below. Within each group the six models differ by at most 0.009. Adjusting all six to the same mix of groups reduces the ratio between the largest and the smallest model average from 1.83 to 1.19, and within Qwen3 the fall from 0.6B to 4B shrinks from 35% to 12%.

2. Both orders are in use (Section 5.5). The submitted version reported that Transformers applies temperature first. As you noted, llama.cpp lets users rearrange its sampler chain. Its standard chain applies temperature last, while vLLM applies it first, and the same reported settings therefore define different distributions in widely used engines. For min-pp the paper supplies the conversion between them. A threshold α\alpha applied before temperature τ\tau gives the same distribution as α1/τ\alpha^{1/\tau} applied after it, and 0.1 in the llama.cpp order equals 0.037 in the vLLM order at τ=0.7\tau = 0.7 when min-pp is the only truncation rule. For top-pp, Proposition 4 shows that no replacement threshold works for every distribution, and the order itself has to be matched or reported.

3. Where the order matters most (Section 5.6). The table uses temperature 0.7 and top-pp 0.95, with both orders driven by the same random numbers.

Task (models)Mean entropy per step (nats)Mean total variation per stepOutputs identical under both orders
HumanEval (LFM2.5, Qwen3-1.7B)0.03 to 0.080.0005 to 0.001472 to 90%
GSM8K (Qwen3 at three sizes)0.11 to 0.310.0016 to 0.005236 to 67%
Open-ended continuation (LFM2.5, Qwen3-1.7B)1.4 to 2.70.020 to 0.0351 to 2%

At temperature 1.2 the per-step value in open-ended text rises to 0.06 to 0.10. The fraction of repeated 4-grams differs between the orders in all six open-ended settings, and each difference stays significant after Holm correction over the 25 added comparisons. For Qwen3-1.7B at temperature 0.7, 19.9% of continuations are degenerate with temperature first against 13.5% with truncation first (difference 6.4 points, 95% interval 3.9 to 8.9). The effect is largest for free text and for temperatures above 1, and it is present at a smaller scale in code and math.

The GSM8K result. You wrote that 1.3 points was more than you would have guessed, and you noted the single model and the lower end of the interval. We take both points. The estimate is 1.34 points (47.12% against 45.78% on 1,319 problems, four samples each, three seeds), and even the lower end of its interval, 0.35, lies above zero. This was the primary comparison of the submitted version. For scale, the accuracy of each order varies by 0.17 and 0.26 points (standard deviation) across the three seeds.

The revision adds eight comparisons below temperature 1, on LFM2.5 and on Qwen3 at 0.6B, 1.7B, and 4B (Table 8). The order that keeps fewer tokens scores higher in 6 of the 8, and in all 8 when answers outside the required format are also read. These shifts reach 0.90 points, and both exceptions belong to one model, Qwen3-1.7B. For LFM2.5 the sign reverses at temperature 1.2 (0.66 points in the other direction), where Proposition 1 reverses which order keeps fewer tokens. Taken one at a time the added shifts have intervals that include zero, and what they contribute is a direction that follows the theory. The 1.34 points of LFM2.5 remains the clearest shift on math, a task where the model is nearly certain at most steps.

Any quantity computed from samples depends on the order, including pass@kk, self-consistency, and comparisons of one model across engines.

We hope this addresses the reservation in your review, and we would welcome your reaction.

Thanks for your detailed response. You've added one helpful clarification, that part of what you offer is a prediction of the size of the effect and of the effect on benchmarks. I think my conclusion remains that this is a well done study into a fairly niche topic that should be better known to NLP researchers. But the effect really is rather small - certainly important to be careful about if you're doing NLP evaluations, but not of major impact for the wider field. I flag to the AC that this is my opinion, not an empirical fact. But my opinion remains that this work would be better suited to a workshop at an NLP conference.

As the discussion period is closing, we will not be able to add further analyses, and we would like to leave a final note. Thank you for the time and effort you put into reviewing our paper and for sharing your view openly in the discussion. Regardless of your final assessment, your comments were very helpful to us. In particular, your question about the broader impact of the result has helped us set the direction of our next study.

Thank you again.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful reading and constructive comments. The reviews describe the problem as clear and the theory as clean, and RfNB notes that the min-pp correction is exact and directly usable. Their shared question is what follows beyond non-commutation. The revised PDF answers it with three Qwen3 models (0.6B, 1.7B, 4B), seven temperatures, and generated text in three tasks. Re-running the original grid reproduces every LFM2.5 and MiniCPM5 number to four decimals.

QuestionNew resultWhere
Does the effect vanish as τ→0\tau\to0 under ties? (RfNB)Corollary 1. The limit is 0 unless more than 1/(1−p0)1/(1-p_0) tokens tie exactly at the top (more than 10 at p0=0.9p_0=0.9). The limit is positive at <0.005% of positions§4.1, §5.1, Table 7
Model size or model design? (ZKBD)The shape of the next-token distribution decides. Reweighting six models to one top-1 mix shrinks the spread across models from 1.83× to 1.19×§5.3, Fig. 1d–f
Why do GSM8K and HumanEval differ? (ZKBD)Per-step entropy. Code 0.03–0.08, GSM8K 0.11–0.31, open text 1.4–2.8 nats per token, and per-step TV follows the same order§5.6, Fig. 3b
Degeneration and operational consequences (RfNB Q3, ZKBD)At τ=0.7\tau=0.7, temperature-first raises degenerate continuations of Qwen3-1.7B from 13.5% to 19.9%, and the direction reverses at τ=1.2\tau=1.2 as Prop. 1 predicts. All repetition differences stay significant after Holm correction§5.6, Fig. 3d–f
pass@k (RfNB)Temperature-first gives higher pass@1 (+0.90 pp, CI [0.23, 1.60]), while truncation-first gives more distinct programs and higher pass@16 estimates, the precision-diversity trade predicted by Prop. 1Fig. 3c, Table 9
More models on GSM8K (mrgG, RfNB)Temperature-first, the tighter truncation, scores higher in 6 of 8 new settings below τ=1\tau=1 and in 8 of 8 when answers given outside the required format are also read. At τ=1.2\tau=1.2, where Prop. 1 reverses the truncation direction, the sign reverses for LFM2.5§5.6, Table 8
Practical relevance (all)llama.cpp applies temperature last, while vLLM and Transformers apply it first. The min-pp setting converts exactly (α↦α1/τ\alpha\mapsto\alpha^{1/\tau}), and Prop. 4 proves that no top-pp conversion exists§5.5

The corrected min-pp pairing gives TV exactly 0 at all 84 (model, τ\tau, α\alpha) settings over 1.77 million positions. All figures are redesigned, and the paper is reorganized for readability, with a worked example in the introduction and one consistent set of terms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.