acceptodds
Under review as a conference paper at ICLR 2027

Let The Model Choose: Selecting Samplers For Diverse Answers

Abstract

Best-of-N selection, pass@k evaluation, and offering users alternatives all draw several answers from one language model. Independent sampling spends part of this budget on repeats. Stratified arithmetic sampling coordinates the answers instead. Each answer decodes one random number drawn from its own slice of the unit interval, so the set spreads over the model’s options while every answer keeps the model’s distribution. Kept for the whole answer, this number is rescaled at every token. Unfortunately, that rescaling operation leads to accumulated precision errors, and in float64 it runs out of precision after a median of 44–79 tokens of a real response. The rest of the text is then chosen by rounding error. This, and similar issues, have prevented more widespread adoption of arithmetic sampling and similar methods. To overcome this flaw, we introduce early release: an answer switches back to ordinary sampling as soon as its prefix interval lies inside its slice. We prove that early release produces exactly the joint distribution of answer sets of infinite-precision stratified arithmetic sampling, and that with eight answers it needs more than float64’s 53 bits with probability below 2−50. In practice it releases after a median of 6–12 tokens and runs 14% faster than always-on arithmetic sampling. It composes with top-p, top-k, min-p, and p-less. Across models from 360M to 70B parameters, early release finds 11.4% more distinct correct answers on CoverageQA over a grid of four truncation rules and four temperatures, raises pass@8 on all 11,313 TriviaQA validation questions from 34.72% to 35.93%, and yields 3.6% more quality-approved meanings on NoveltyBench, with no detectable change in the quality of individual answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, samplers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KU5gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes "Early Release" to improve stratified arithmetic sampling for LLM. It switches to normal sampling once the generated prefix is fully inside its assigned stratum. The authors provide theoretical guarantees and show improved answer diversity and coverage across different models and tasks.

Critical strengths
  1. Simple and effective method. I like this type of work that makes a small but clear improvement. The problem is well motivated, and the proposed solution is simple. It does not require any model training or modification.
  2. Good theoretical support. The authors prove that "Early Release" preserves the joint distribution of stratified arithmetic sampling under exact arithmetic. The proof gives a clear reason why the method works, rather than only showing empirical improvements.
  3. Comprehensive experiments. The experiments cover many models, sampling settings, and benchmarks. The authors also provide ablation studies, runtime comparisons, and statistical analysis. I appreciate that they report negative results on reasoning and coding tasks.
  4. Clear writing. Overall, the paper is well written and easy to follow. The motivation, method, and results are presented clearly.
Critical weaknesses
  1. Gap in model sizes. There is a large gap between 7B and 70B in the experiments. Adding 14B or 32B models would make the results more complete.
  2. Numerical precision. The proof assumes exact arithmetic, but the implementation uses float64 to update I(h). It is unclear whether rounding errors near stratum boundaries could cause incorrect early release.
  3. Lack of analysis on where diversity appears. The method shows limited gains on math, code, and reasoning tasks. It would be useful to analyze when meaningful diversity appears during generation, and how this relates to the release position and the improvement.
  4. Limited benefit over simpler methods. Figure 6 shows that coordinating only the first token already gives most of the improvement. The additional practical benefit of the full method is not very clear.
  5. Runtime overhead. Early Release is still around 12% slower than independent sampling on average. A comparison under the same inference time budget would be useful.
  6. Figure quality. The figures could be improved, especially Figure 1. Some elements are small, and the main idea could be shown more clearly.
Questions for authors

Please see weaknesses section, and also some additional questions here:

  1. Follow up on weakness #3: Is there a relationship between the position of semantic differences, the release position, and the final improvement?
  2. Follow up on weakness #4: When is coordinating more tokens actually useful?
  3. How sensitive are the results to token ordering in arithmetic sampling? Would different orderings change the diversity improvement?
  4. Could the authors discuss when Early Release should be preferred over Lattice Sampling in practice?
Reviewer_MzKfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Early Release, a stratified arithmetic sampling method for generating diverse answer sets. This method coordinates sampling during the early tokens and switches back to ordinary sampling once the prefix interval falls entirely within the corresponding slice, preserving the joint distribution of infinite-precision stratified arithmetic sampling while avoiding floating-point exhaustion. Experiments across multiple models and tasks show improved candidate coverage and modest gains in TriviaQA pass@8 and semantic diversity, with little change in individual answer quality.

Critical strengths
  1. The motivation is clear and addresses the numerical instability and computational cost of always-on arithmetic sampling.

  2. The experiments cover multiple model scales, truncation strategies, and evaluation tasks.

Critical weaknesses
  1. The gains are relatively modest: TriviaQA pass@8 improves by only 1.21%, while NoveltyBench semantic coverage increases by 3.6%.

  2. The benefits are concentrated on tasks where answers diverge early; there is little improvement on mathematical reasoning, code generation, and other long-form reasoning tasks.

  3. CoverageQA contains only 145 questions and relies on predefined accepted answers, which may limit the generality of the conclusions.

  4. The comparison with stronger coordinated or quasi-random sampling methods, such as QuasiMoTTo, is insufficient.

  5. The NoveltyBench results depend on LLM judges and a semantic classifier, but the paper provides limited analysis of evaluator bias and robustness.

Questions for authors
  1. How does Early Release compare with QuasiMoTTo and other coordinated sampling methods under matched compute budgets?

  2. What are the latency and throughput costs for larger KK, longer contexts, and realistic serving workloads?

  3. Can the authors characterize which task or answer properties determine whether Early Release provides meaningful gains?

2 comments

Thanks for the rebuttal and additional experiments. Most of my concerns are addressed. The remaining limitations primarily lie in the relatively modest absolute performance gains for certain tasks, as well as the relatively high computational overhead for short outputs and small models. I therefore maintain my current rating.

Thank you for your follow up. In the last comment, I clearly state that most of my concerns are addressed and there are two remaining limitations. I think this paper will be more valuable if the authors can solve the remaining limitations. Currently I will maintain the positive rating.

Reviewer_KgCXOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper diagnoses that always-on stratified arithmetic sampling exhausts float64 precision after a median of 44–79 tokens, leaving later tokens to rounding error. It introduces "early release": once an answer's prefix interval lies inside its assigned slice of [0,1), decoding switches to fresh independent draws. The authors prove this exactly preserves the joint answer-set distribution of infinite-precision stratified sampling (Theorem 1) with a model-independent bit bound (Prop. 3), and show modest but statistically robust diversity gains across models from 360M to 70B.

Critical strengths

Clean theory: Theorem 1 gives both exactness and necessity of the containment rule, and Proposition 3 bounds coordination length (≤2^-50 overflow risk at K=8), with correct proofs in Appendix A.

A genuinely useful diagnosis: float64 exhaustion after a median of 44–79 tokens and 96% response flips under 10^-4 logit perturbation (Sec. 2.2, App. B.1) document a real, underappreciated flaw.

Critical weaknesses

Modest effect sizes: TriviaQA pass@8 gains only +1.21 points (34.72→35.93), CoverageQA 0.157 distinct answers, NoveltyBench 3.6%; whether this matters for best-of-N practice—where a verifier must still find the correct candidate—deserves franker discussion.

Benefit scope is narrower than the framing: Table 1 shows no detectable gain on MATH-500, GSM8K, IFEval, HumanEval+/MBPP+, or reasoning models, since release happens on formulaic opening tokens; the abstract should state this boundary up front.

Missing empirical comparison to QuasiMoTTo's other quasi-random designs (Li et al., 2026); notably, the lattice baseline slightly beats early release on the CoverageQA average (1.564 vs. 1.528, Sec. 4.1), so the case for strata rests on the pass@K guarantee rather than empirical dominance.

NoveltyBench evidence is thin: 100 prompts, three pooled models, LLM-judge approval, and no gain on the 0.5B model.

Questions for authors

On CoverageQA the lattice attains more distinct answers on average; can you report lattice vs. early release per task with the same intervals, and characterize events where the lattice's shared offset hurts (App. A.5) empirically?

Does early release compose with QuasiMoTTo's Sobol/lattice constructions, or is it tied to independent strata?

The pass@K guarantee (Eq. 1) covers fixed events; can you bound or measure the effect on judge-dependent metrics, which the limitation section excludes?

For reasoning models, release occurs after ~23 formulaic tokens; would later re-coordination points (e.g., per-step strata) recover gains there, as the conclusion hints?

Can you share per-setting overhead numbers so practitioners can decide when the 12% average cost is worth the diversity gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.