- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how the sampling distribution of LLMs changes under some operation over this distribution. Specifically, the authors focus on temperature scaling and truncation operation. They show that in most cases these operations do not commute. They then conduct some experiments to quantify how changing the orders of these operations impact the resulting distribution.
Critical strengths
The problem studied is clear and well defined. The overall writing is very clear and pleasant to read. The maths seems carefully conducted (though I did not check in details). Conclusions are clear.
Critical weaknesses
I have two main reserves for this paper. The first is that I am unsure that pointing out that truncation and scaling do not permute is a particularly strong contribution (though this is done cleanly here). The second is about the experiments.
- authors used several but rather "less known" models. I wonder what are the justifications for using these models rather than others.
- Then, I think the experiments overall support the claims of the paper. However, since most results are consequential from the above analysis, I could have appreciated other "empirical" analysis. For instance:
- Do authors have insights about whether such discrepancies between distributions change with model size? Or is it rather linked to the model's internal design?
- How does the discrepancies change with the task? For instance, on Figure 2 in appendix F, we see rather distinctive curves on GSM8k vs HumanEval. I wonder if authors have any insight about this.
- A more refined analysis of the concrete implications could also be valuable. Table 2 from the main text already hints at this, but most of the paper is about "theoretical distributional measures" but more operational consequences, or model's sensitivity to this could be appreciated.
Overall, the paper supports its main claims. But since the main claim (temperature scaling and truncation do not commute) is not especially strong, I could have expected the paper to be more ambitious in exposing the practical implications of this. In the current form, it feels a bit shallow.
Questions for authors
See weaknesses.
5 comments
Assessment confirmed
Dear authors, Thank you for your answer.
I also read the review of mrgG and it seems we largely agree on the proposed contribution.
The rebuttal does not clearly answer my concerns. First, I want to mention the rebuttal is much less clear than the paper, with lots of technical jargon and verbose conclusions that are very hard to process.
For instance:
- span next-token distribution shape (mean top-1 probability 0.43–0.61), the quantity the theory depends on I find this wording really unclear and does not convincingly answer my question.
- Model size or internal design? (§5.3, Fig. 1d–f). The shape of the next-token distribution decides. I do not understand this conclusion.
- Reweighting each model to one top-1 mix removes most of the spread I do not understand what it means, nor am I sure you have enough empirical evidences to make such a claim.
- Larger models place more predictions where the order cannot matter, since whenever the top token alone carries the two orders coincide exactly. Same issue.
Regarding the empirical insights, I do not clearly better understand the practical implications of the studied problem.
I decide to maintain my rating.
Thank you for reading our response of November 7 and for telling us directly where it fell short. To fit all our answers into one comment under the character limit, we compressed them until they depended on shorthand that we never defined. That was our mistake, since we could have used several comments, and we apologize for the time it cost you. We answer again in two comments and begin with the sentences you quoted.
Short answers.
- Model size or model design? Within the Qwen3 family the effect falls by 35% from 0.6B to 4B parameters. Most of that fall comes from one measurable property, how often the model is confident about the next token.
- Why these models? The first three differ in that property and come from three developers. The revision adds Qwen3 at three sizes to separate size from design.
- Why do GSM8K and HumanEval differ? The model is uncertain at more steps in math than in code.
- What should a user do? Report the order together with the settings, convert min- thresholds with one formula, and for top- set the same order in the engine, because no change of the top- value makes the two orders agree.
The setting, in one example. With next-token probabilities 0.6, 0.3, and 0.1, temperature 0.5, and top- 0.7, applying top- first leaves two tokens and samples the second one 20% of the time, while applying temperature first leaves one token and never samples the second. Reported settings do not say which order was used. Min- is a second common rule, which keeps the tokens whose probability is at least a set fraction of the top token's probability.
What we measure. For each next-token prediction we compute the total variation distance between the sampling distributions of the two orders. A distance of 0.03 means that the two orders must choose different tokens in at least 3 of every 100 draws. The numbers below use temperature 0.7 and top- 0.95 on the WikiText-103 test set.
"The shape of the next-token distribution decides." By shape we meant how much probability the model gives to its top candidate, which we call its confidence at that position. We grouped all positions by this probability. Each cell is the average distance between the two orders.
| Probability of the top candidate | MiniCPM5 | LFM2.5 | Qwen3-0.6B | Granite-4.1 | Qwen3-1.7B | Qwen3-4B |
|---|---|---|---|---|---|---|
| below 20% | 0.0468 | 0.0464 | 0.0452 | 0.0440 | 0.0429 | 0.0417 |
| 20 to 40% | 0.0437 | 0.0421 | 0.0414 | 0.0397 | 0.0381 | 0.0369 |
| 40 to 60% | 0.0371 | 0.0352 | 0.0345 | 0.0309 | 0.0296 | 0.0280 |
| 60 to 80% | 0.0291 | 0.0285 | 0.0271 | 0.0241 | 0.0232 | 0.0222 |
| 80 to 90% | 0.0210 | 0.0220 | 0.0215 | 0.0215 | 0.0214 | 0.0213 |
| 90 to 95% | 0.0083 | 0.0089 | 0.0101 | 0.0123 | 0.0125 | 0.0130 |
| 95% or more | 0 | 0 | 0 | 0 | 0 | 0 |
Down each column the distance falls from about 0.045 to zero. Across each row the six models, from four developers and from 0.6 to 4 billion parameters, differ by at most 0.009. Confidence at a position tells much more about the order effect than the identity of the model does. What remains within a row is small and orderly, with the larger Qwen3 models slightly lower in the first four rows.
"Predictions where the order cannot matter." The last row is zero by a short argument and needs no data. If the top token has probability 97% and top- is 0.95, then at any temperature of 1 or below top- keeps only that token whichever operation comes first. Models differ in how often they are in this row. It holds 9% of positions for MiniCPM5 and LFM2.5, 15% for Qwen3-0.6B, 23% for Granite-4.1, 26% for Qwen3-1.7B, and 29% for Qwen3-4B. Within Qwen3 the share grows with size.
"Reweighting each model to one top-1 mix." This was shorthand for a standard adjustment, the one used when death rates of two countries are compared after adjusting for age structure. We recomputed the average of each model as if all six had the same share of positions in each row. The ratio between the largest and the smallest model average falls from 1.83 to 1.19, and within Qwen3 the fall from 0.6B to 4B shrinks from 35% to 12%. Most of the difference between models therefore comes from how often each model is confident. Min- with one threshold in both orders shows the same pattern (2.02 to 1.14, Section 5.3).
What the evidence covers. The table covers 1.77 million predictions, which is every position of the test set for six models from four developers. The last row is proved for any model. The other rows are measured at temperature 0.7 and top- 0.95 on models of up to 4 billion parameters, with the size trend read from the three Qwen3 models, which share one design. The measured claim is that confidence explains most of the difference between these models, and the theory holds for any model of any size.
Thank you for the review and for finding the problem well defined and the writing clear. We followed each of your suggestions, and the resulting analyses are now the core of the revised results.
Choice of models. We chose the original three models from three developers to span next-token distribution shape (mean top-1 probability 0.43–0.61), the quantity the theory depends on. To separate size from design, the revision adds Qwen3 0.6B, 1.7B, and 4B, which are widely used and share one architecture family, training recipe, and tokenizer.
Model size or internal design? (§5.3, Fig. 1d–f). The shape of the next-token distribution decides. Within Qwen3 the top-p TV falls with size, yet at a fixed top-1 probability all six models give nearly the same TV (0.0210–0.0220 for top-1 in [0.8, 0.9)). Reweighting each model to one top-1 mix removes most of the spread (τ=0.7, p0=0.95):
Larger models place more predictions where the order cannot matter, since whenever the top token alone carries p0 the two orders coincide exactly. A top-1 histogram from one forward pass therefore predicts the effect for a model and corpus.
Why do GSM8K and HumanEval differ? (§5.6, Fig. 3b). The answer is per-step entropy along the generation:
The ranking can be read from the untransformed model in advance.
Operational consequences.
Contribution. Non-commutation is the starting point of the paper, and its contribution is the exact boundary of order independence: a model- and vocabulary-free correction for min-p, a proof that no threshold correction exists for top-p, a bound within 1.4% of tight below unit temperature, and the condition under which the effect vanishes. The new results show what sets its size and where it changes generated text.
We hope the revision addresses your questions, and we would be glad to discuss any of them further.