acceptodds
Under review as a conference paper at ICLR 2027

Sampler Gains Shrink to Noise Under Equal Tuning Budgets: The Winner's Curse in LLM Decoding Evaluation

Abstract

Truncation samplers such as top-, min- and typical sampling are often reported to beat one another by sweeping the proposed sampler, reporting its best score and comparing it with untuned defaults. We show that this practice carries a quantifiable winner's curse. Modeling tuning as random search with budget over paired evaluation noise, we express the inflation of best-of- reporting and the spurious gap between identical methods tuned with unequal budgets, and show which part of a tuned-vs-untuned gain a dev/test split removes; on seed-replicate runs, whose truth is known, measured inflation runs at – the formulas. Our equal-budget audit (EBET) gives every sampler family the same budget, selects on dev items and reports on held-out ones, across eleven model–task tiers on GSM8K and MATH-500 (1,218,657 generations in all). On these benchmarks temperature, top-, top- and typical are indistinguishable at their tuned optima: equivalence tests on the Qwen tiers exclude pooled differences of points or more (smaller ones are not resolved), and no family beats greedy past the protocol's resolution floor. Under our min- prior, which puts most of its temperatures above its rivals' caps, min- trails top- in all eight Qwen tiers; redrawn within their range, or with paired to as its guidelines advise, it ties ( and pooled), though neither arm excludes a deficit as large as the original. Re-analyzed in a stylized published style (best of 24 against an untuned default), the same records show gains of to points in every tier, by our estimate mostly selection bias. All code and per-item generation records will be made public at https://github.com/xxx/xxx upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
llm decoding, sampling methods, evaluation protocol, winner's curse, selection bias, hyperparameter tuning, equal-budget comparison

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ppDJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles an important problem that is unfortunately common across ML papers, the mismatch between what's reported in the paper and what's statistically valid, as often what's reported is the result of additional hyper parameter optimisation that the comparison baselines do not receive. The paper particularly focuses on the case of sampling procedures, including the common top-k, min p etc.

Critical strengths

I believe it's an important problem and good match for the datasets and benchmarks track, this is a very common problem in ML papers. The proposed methodological solutions/corrections are good and would help identify actual differences in methods.

Critical weaknesses

I mainly just have an issue with the more limited empirical analysis performed for the datasets/benchmark tracks.

Specifically:

  • The limited evaluations. Only two datasets and three model families are used.
  • Exchangeability vs Expected accuracy: More discussion on these is warranted, and what affect that has on the conclusions.
  • The Eight-tier sign tests when the results are not independent

The two datasets limitation is quite important, as in the min-p papers, the authors represent min-p as improving diversity of generated answers, so looking just at accuracy on two math datasets is unlikely to get at the differences or the common use of such samplers, so perhaps understates the differences in this regard.

Smaller point, Table 1 is quite poorly presented and would benefit from reformatting.

Questions for authors

Please see the weaknesses for questions. But the key is that expanded comparisons, and considering metrics beyond just expected accuracy is important here. For the benchmarks track, this warrants more datasets, more metrics, and more comparisons.

Reviewer_eTCCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the winner’s curse in LLM sampling evaluation: selecting and reporting the best configuration can inflate apparent gains, particularly when tuning budgets differ across methods. It develops a theoretical account of this bias, validates its predictions through seed-replicate experiments, and introduces EBET, an equal-budget protocol with held-out evaluation.

Critical strengths
  1. I find the research question interesting, as it approaches sampling research from an angle I had not previously considered. It may help us better understand sampling's stochastic nature.
Critical weaknesses
  1. I believe the writing needs substantial improvement. Starting in the introduction, the exposition quickly becomes dense and difficult to follow, with numerous definitions and theoretical results presented in an opaque manner. Understanding the paper requires repeatedly moving back and forth between different sections. Some phrasing also feels awkward and unnatural, such as “We do not claim samplers never matter... Our claim is about evaluation...” (lines 107–109). I strongly recommend that the authors revise the exposition with the reader’s perspective in mind.

  2. The propositions in the theoretical section are presented without any explanation of their motivation, making it difficult to understand their purpose. The lengthy blocks of text in the appendix are also extremely difficult to read.

  3. Regarding the experimental setup, while the study investigates randomness associated with sampling configurations, it does not appear to eliminate randomness arising from the dataset itself or even from the dev/test split. The experimental evaluation is also too limited. The model family used in the experiments, Qwen2.5, is quite dated, and it remains unclear whether frontier models have already substantially mitigated this issue, particularly given that Qwen2.5 has not demonstrated long-cot reasoning capabilities.

  4. Eq(2) and Table 1 extend beyond the right margin. I am unsure whether this violates the page-limit rules. In any case, I strongly recommend that the authors make a serious effort to address the paper’s readability issues.

Questions for authors

The paper appears to cite a substantial number of studies from the early 2000s to support its arguments, particularly in the related work section. However, these studies do not seem closely related to sampling. I am unsure whether these citations are necessary. Could the authors provide a convincing justification for their inclusion?

Reviewer_4oZsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to resolve a prevalent dispute in the principled evaluation of truncation samplers such as top-p, min-p and typical sampling commonly employed in text generation from LLMs. The paper can be seen as a logical extension to the prior work by Schaeffer et al 2025 which shows that min-p sampling does not surpass other common sampling methods when hyperparameter tuning budget is controlled for. This raises serious concerns about the perceived gains for various sampler purported in the published literature. The authors extend the line of work and conduct an exhaustive study comparing 3 common truncation samplers on Math tasks for a few different models. The paper also tries to build an interpretive theoretical model to explain the phenomenon, though several assumptions in the theory are stronger than what can be expected to hold in practice. In that light, the theory is merely interpretive than predictive and should be read with that lens.

Critical strengths
  • S1: The paper sheds light on an important but overlooked aspect of truncation sampler evaluations.
  • S2: The theoretical model provides a useful mental model to interpret the results. But several parts of the model are not very clearly justified. For e.g., equal correlation, etc.
Critical weaknesses
  • W1: It is hard to pin down the central novel claims in the paper. My current reading is that the paper lacks novelty - the main propositions formalize standard facts about selection bias from a group of noisy estimates. The held-out test set removes selection bias but not tuning benefit is also obvious. The framing around this could be sharpened and what the paper contributes over Schaeffer et al could be made clearer.

  • W2: The experiment section is thin and the paper primarily tests on Math benchmarks. IMHO a math benchmark would not isolate all the benefits from various kinds of truncation sampling algorithms. One of the primary advantages of truncation samplers is to avoid degenerate text and just checking for final answer correctness does not capture this angle. The overall claims would be strengthened by also including more comprehensive testing on non-math benchmarks. The 30-prompt creative-writing experiment is very limited and inconclusive.

  • W3: I found the writing and presentation difficult to follow. Several sections have dense set of numbers before describing the higher level conclusion they support. A lot of the writing seems rushed and unwieldy.

Questions for authors
  • What does the paper establish beyond prior work and Schaeffer et al. (2025)? Why is equal correlated gaussian a good theoretical model here? Why not use an alternate model e.g., a simple linear regression model? What justifies this choice?
  • How were the hyperparameter ranges or distributions for each sampler chosen? Since min-p’s result changes when its search distribution changes, how can we tell whether the observed differences come from the samplers themselves rather than from the particular hyperparameter ranges used for tuning?
  • I am unclear about what the permutation test establishes. Can it reject simply because one family’s search prior contains more poor configurations, even when the configurations selected after tuning perform similarly? If so, how should I relate these p-values to the paper’s main comparison of tuned performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.