- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes truncating with the model's original probabilities first and then sampling uniformly among the kept tokens, which is the same as sampling at infinite temperature. Global KL* picks the top-k set whose uniform distribution is closest to the model in KL divergence, and local KL* is a greedy version that stops at the first increase of that score. A third rule, KL-budget, flattens a min-p set as much as a fixed KL budget allows. The main experiment uses the full GSM8K test set with three small models, with comparisons fixed in advance and Bonferroni-adjusted intervals. Smaller studies cover five other tasks, larger models, a reasoning model and creative writing.
Critical strengths
The rule is cheap on CPU and needs no tuning. I also liked the result that the best uniform set is always a top-k set. The evaluation is done carefully. The protocol is fixed before the runs, all 1,319 GSM8K questions are used, the losses are reported, and all generations are released. The high temperature study is a clear result, since temperature-first samplers collapse there and truncation-first ones don't, although it uses fixed settings rather than a tuned comparison.
Critical weaknesses
The main claim of this paper rests on not finding significant differences. With so many corrected comparisons the intervals are wide, so "no significant loss" does not show that KL* matches the recommended settings, and the paper itself admits this does not establish equivalence. Where differences do show up, they don't always favor KL*. P-less is clearly more accurate per answer on two models, the tuned baselines have the best pass@8 on every main model, and KL-budget loses far more often than it wins. Even taken at face value, the differences between samplers are small, so I think the contribution is too limited for ICLR.
I found the evidence of this paper narrow. The main models are small 2024 checkpoints on GSM8K, the larger models are only run on 32 to 64 questions, and the paper notes that a ranking from one of its subset studies did not hold on the full test set. With a reasoning model, all samplers end up within about a point of each other.
I also found the paper hard to read, because the main text depends too much on a very long appendix.
I did not find the score itself well motivated. The paper does not explain why the draw should be uniform or why this KL objective is the right one, and its own analysis shows that without the uniform restriction the same objective just gives ordinary temperature sampling. The results also suggest the score does not matter much, since the local rule, which can be far from the optimum, does at least as well as the exact global rule.
Finally, cost is only measured for the sampler alone on CPU. There is no end-to-end timing, and the method needs a new GPU kernel for engines like vLLM, which was not tested.
Questions for authors
Could you run at least one of the larger models (for example Qwen3.8-27B with thinking off) on the full GSM8K test set instead of a small subset?
If the exact global minimum of the score does not beat the local rule, why is this score the right objective?
Can you report end-to-end generation time, and the sampler cost in a batched GPU engine such as vLLM?