acceptodds
Under review as a conference paper at ICLR 2027

Sampling at Infinite Temperature: KL* and KL-Budget Decoding

Abstract

Applying temperature before truncation lets unlikely tokens into the candidate set, and a high temperature can then derail generation. We reverse the order. The model’s original probabilities choose the candidates, and exploration happens only among them. At infinite temperature every candidate is equally likely, so the candidate set alone decides what is sampled. Global KL* picks the candidate set whose uniform draw is closest to the model in KL divergence. We also study local KL*, a greedy variant, and KL-budget, which flattens a min-p candidate set within a fixed KL allowance. On the complete GSM8K test set, both KL* rules match each model’s recommended sampling settings without any tuning, and global KL* has no significant loss in pass@8 or majority vote against any standard default sampler. Across six tasks, the KL* rules gain two to three times as often as they lose against these defaults. On real next-token distributions they explore below the top token while rarely reaching the tail, and at high temperature truncating first prevents the collapse of high-temperature sampling. The rules remain competitive in creative writing and with reasoning models. We release all generations and analysis code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Infinite Temperature, Temperature, Sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VoHfOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes truncating with the model's original probabilities first and then sampling uniformly among the kept tokens, which is the same as sampling at infinite temperature. Global KL* picks the top-k set whose uniform distribution is closest to the model in KL divergence, and local KL* is a greedy version that stops at the first increase of that score. A third rule, KL-budget, flattens a min-p set as much as a fixed KL budget allows. The main experiment uses the full GSM8K test set with three small models, with comparisons fixed in advance and Bonferroni-adjusted intervals. Smaller studies cover five other tasks, larger models, a reasoning model and creative writing.

Critical strengths

The rule is cheap on CPU and needs no tuning. I also liked the result that the best uniform set is always a top-k set. The evaluation is done carefully. The protocol is fixed before the runs, all 1,319 GSM8K questions are used, the losses are reported, and all generations are released. The high temperature study is a clear result, since temperature-first samplers collapse there and truncation-first ones don't, although it uses fixed settings rather than a tuned comparison.

Critical weaknesses

The main claim of this paper rests on not finding significant differences. With so many corrected comparisons the intervals are wide, so "no significant loss" does not show that KL* matches the recommended settings, and the paper itself admits this does not establish equivalence. Where differences do show up, they don't always favor KL*. P-less is clearly more accurate per answer on two models, the tuned baselines have the best pass@8 on every main model, and KL-budget loses far more often than it wins. Even taken at face value, the differences between samplers are small, so I think the contribution is too limited for ICLR.

I found the evidence of this paper narrow. The main models are small 2024 checkpoints on GSM8K, the larger models are only run on 32 to 64 questions, and the paper notes that a ranking from one of its subset studies did not hold on the full test set. With a reasoning model, all samplers end up within about a point of each other.

I also found the paper hard to read, because the main text depends too much on a very long appendix.

I did not find the score itself well motivated. The paper does not explain why the draw should be uniform or why this KL objective is the right one, and its own analysis shows that without the uniform restriction the same objective just gives ordinary temperature sampling. The results also suggest the score does not matter much, since the local rule, which can be far from the optimum, does at least as well as the exact global rule.

Finally, cost is only measured for the sampler alone on CPU. There is no end-to-end timing, and the method needs a new GPU kernel for engines like vLLM, which was not tested.

Questions for authors

Could you run at least one of the larger models (for example Qwen3.8-27B with thinking off) on the full GSM8K test set instead of a small subset?

If the exact global minimum of the score does not beat the local rule, why is this score the right objective?

Can you report end-to-end generation time, and the sampler cost in a batched GPU engine such as vLLM?

Reviewer_nUxEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies decoding that selects candidates using the original model probabilities before flattening their weights. Global KL* selects the support whose uniform distribution minimizes KL divergence to the model, while local KL* uses a greedy stopping rule. The paper also studies KL-budget decoding and compares these approaches with standard sampling configurations across reasoning and generation tasks.

Critical strengths

S1: Global KL* is a simple, well-defined rule that removes the need to choose a sampling temperature or truncation threshold. The distinction between the exact global solution and the potentially inferior local rule provides useful theoretical guidance.

S2: The experiments make a credible case for KL* as a practical default. Comparing against model-recommended settings is relevant to users who cannot tune a sampler for each task. The analysis of candidate support also helps explain how uniform sampling can preserve plausible alternatives without frequently selecting tail tokens. Competitive performance without tuning is the main practical contribution supporting acceptance.

Critical weaknesses

W1: The evidence supports competitiveness more clearly than equivalence or general superiority. In Figure 4, confidence intervals containing zero establish that a difference was not detected; they do not establish that meaningful losses have been ruled out. In addition, truncation before temperature and the KL-budget formulation build on acknowledged prior work. The contribution therefore depends mainly on the specific KL* support rule and the demonstrated value of avoiding tuning.

Questions for authors

Q1: For the main comparisons in Figure 4, what accuracy losses remain compatible with the reported intervals? Please distinguish the evidence for “no detected difference” from any stronger claim that KL* matches the recommended settings within a practically meaningful tolerance.

Reviewer_amtUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a truncation-first view of high-temperature sampling. The model’s original probabilities first select a candidate set, after which exploration is performed only within that set; at infinite temperature the surviving candidates are sampled uniformly. Global KL* chooses the top-k set whose uniform distribution minimizes KL divergence to the original distribution, local KL* greedily stops when adding the next token increases this divergence, and KL-budget flattens a min-p candidate set subject to a fixed KL budget. The paper gives exact and efficient procedures and proofs for the KL* rules, then evaluates them against common samplers on GSM8K, MATH-500, TriviaQA, CommonsenseQA, QASC, reasoning-model settings, real next-token statistics, and creative writing. On the reported default-setting experiments, KL* methods are generally competitive with model-card/default samplers without tuning, while retaining exploration and avoiding the collapse observed when temperature is applied before truncation.

Critical strengths
  1. A simple, principled, and tuning-free sampler. Global KL* turns the vague goal of “explore without entering the implausible tail” into an exact one-dimensional optimization over top-k sets. It is computationally cheap and has a clear interpretation, which makes the contribution practically usable.

  2. Exceptionally broad empirical and formal validation for a decoding rule. The paper includes complete or large evaluation splits, multiple task types and model scales, high-temperature order studies, reasoning-model scopes, real next-token statistics, timing measurements, and appendices that expose where the methods lose. This substantially strengthens confidence that the observed behavior is not a single benchmark artifact.

Critical weaknesses
  1. The empirical advantage is not consistently decisive and the scope is narrower than the headline framing. The paper often shows competitiveness or ties rather than a robust improvement over strong defaults; it explicitly notes that global and local KL* are not consistently ordered, and conservative baselines can be better for single-answer accuracy. The main contribution is therefore best understood as a principled robust default for a particular sampling regime, not a generally superior sampler.

  2. Dependence on automated evaluation and a single serving stack. Creative-writing quality is assessed primarily by an automated Gemma judge, while the decoding implementation and timing results come from one inference engine and quantized checkpoints. The reported writing wins and overhead may not transfer to human preferences, other judges, full-precision logits, or other sampler implementations. This is especially relevant because small differences are used to support practical recommendations.

Questions for authors
  1. Do the main task and writing conclusions persist across a second inference implementation and full-precision logits, especially for the high-temperature/order-collapse experiments and the timing comparisons?

  2. For the creative-writing claims, how much do the conclusions change under human evaluation or an independent judge family? Please report per-checkpoint uncertainty and clarify which comparisons remain significant after accounting for all sampler/model contrasts.

  3. Please provide a single table specifying the number of hypotheses in each aggregate “gain versus loss” statistic, the unit of resampling, and the exact multiplicity correction. Which headline conclusions remain if comparisons are aggregated first per task and model rather than over all individual contrasts?

  4. The method is described as tuning-free, but KL-budget requires a KL allowance and min-p base set. How should practitioners select these values without development tuning, and how sensitive are the recommendations to them outside the reported settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.