Sampler Gains Shrink to Noise Under Equal Tuning Budgets: The Winner's Curse in LLM Decoding Evaluation

Truncation samplers such as top-$p$, min-$p$ and typical sampling are often reported to beat one another by sweeping the proposed sampler, reporting its best score and comparing it with untuned defaults. We show that this practice carries a quantifiable winner's curse. Modeling tuning as random search with budget $K$ over paired evaluation noise, we express the inflation of best-of-$K$ reporting and the spurious gap between identical methods tuned with unequal budgets, and show which part of a tuned-vs-untuned gain a dev/test split removes; on seed-replicate runs, whose truth is known, measured inflation runs at $0.81$–$1.10\times$ the formulas. Our equal-budget audit (EBET) gives every sampler family the same budget, selects on dev items and reports on held-out ones, across eleven model–task tiers on GSM8K and MATH-500 (1,218,657 generations in all). On these benchmarks temperature, top-$k$, top-$p$ and typical are indistinguishable at their tuned optima: equivalence tests on the Qwen tiers exclude pooled differences of $3.5$ points or more (smaller ones are not resolved), and no family beats greedy past the protocol's resolution floor. Under our min-$p$ prior, which puts most of its temperatures above its rivals' caps, min-$p$ trails top-$p$ in all eight Qwen tiers; redrawn within their range, or with $p$ paired to $T$ as its guidelines advise, it ties ($-.001$ and $.000$ pooled), though neither arm excludes a deficit as large as the original. Re-analyzed in a stylized published style (best of 24 against an untuned default), the same records show gains of $+1.4$ to $+8.8$ points in every tier, by our estimate mostly selection bias. All code and per-item generation records will be made public at https://github.com/xxx/xxx upon acceptance.