acceptodds
Under review as a conference paper at ICLR 2027

Sampling More, Getting Less: A Token-Level Diagnostic for LLM Generation Diversity

Abstract

Diversity is essential for language-model applications ranging from creative generation to scientific discovery. Yet modern LLMs lack diversity and increasing the sampling temperature can degrade text quality before sufficient diversity is achieved. Existing decoding methods address this trade-off through token-level filtering. While they provide heuristics about how valid tokens appear in the next-token distribution, their measurements happen at the sequence level. This leaves a gap between where the methods are actually proposed and where they are being evaluated. We introduce a framework for diagnosing and quantifying the validity-diversity trade-off directly at the token level. Our approach combines tasks with verifiable output spaces and an automated procedure for open-ended tasks. Using our framework, we identify two failures in LLMs: First, order calibration: valid tokens are not reliably ranked above invalid tokens. No decoding method based on top-token filtering can recover enough valid tokens without including many invalid tokens. Second, shape calibration: probability mass is overly concentrated only on a few valid tokens while having a heavy tail of mixed valid and invalid tokens, so maintaining high diversity will inevitably expose the invalid tokens in the tail. We formalize both mechanisms and show that local failures compound across decoding steps, producing strong sequence-level losses in diversity. Across 14 language models spanning multiple families and scales, we find that diversity collapse is not merely a limitation of particular sampling heuristics, but a consequence of order and shape miscalibration in the LLM distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diversity Collapse, Probability Calibration, LLM Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_q4c1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies order and shape calibration. Theory and experiments link token-level errors to limited output diversity. The exact measurements make this a useful diagnostic study. However, uncertainty in the token labels limits support for the broader conclusions. The uniform oracle also leaves the cause of its gains unclear. These concerns affect the main interpretation of the results and support my weak rejection.

Critical strengths

The controlled tasks provide strong evidence. Section 5.2 uses tasks with known sets of valid outputs. The authors can therefore calculate validity and diversity exactly. These measurements avoid uncertainty from judgments about text quality. The results support the proposed causes of diversity loss in these tasks.

Critical weaknesses

A token can have a valid continuation even when its greedy completion fails. Thus, greedy decoding can mark a valid token as invalid. Appendix A.3 includes useful checks with additional continuations. In Table 3, recall at precision 0.8 increases from 0.09 with greedy decoding to 0.34 with ten sampled continuations. The trade-off remains, but its measured size depends on the procedure used to assign labels. This uncertainty weakens broad claims about open-ended generation.

The oracle in Section 4.4 selects tokens labeled valid and gives them equal probability. It therefore changes both token selection and relative probabilities. Its gains do not isolate the effect of token order errors. Table 1 selects the other methods at validity of at least 0.8. However, the table does not report each method’s measured validity. The comparison therefore does not establish whether the diversity gains remain at the same validity rate.

Questions for authors

What fraction of tokens labeled invalid by greedy decoding receive a valid label with ten sampled continuations? How do these label changes affect the estimated limits of rank-based filtering?

Do the Table 1 gains remain when the methods have the same measured validity rate? Please report each method’s validity. Please also test an oracle that preserves the model’s relative probabilities among tokens labeled valid.

Reviewer_26wcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the validity-diversity trade-off issue in LLM sampling. Increasing temperature can reduce validity before producing sufficiently diverse outputs. The authors propose connecting sequence-level quality and diversity to the token-level decisions made by truncation samplers. It introduces diagnostics for the ranking and probability mass of valid alternatives, combining exactly enumerable tasks with judged continuations in open-ended generation.

Critical strengths
  • The motivation identifies a useful evaluation gap. Sequence-level diversity scores can reveal repetitive outputs without showing which local decisions exclude valid alternatives. Inspecting token validity and probability concentration could help distinguish a poorly chosen sampler from limitations of the underlying distribution. This diagnostic objective is definitely interesting and worthwhile even without proposing a new decoder.
  • The precision–recall perspective provides insights into what typical truncation based sampling may miss out. It distinguishes retaining a clean set of candidate tokens from preserving access to a broad range of valid completions. When valid and invalid tokens are interleaved in probability rank, these objectives conflict even with an optimally chosen cutoff. This helps explain why tuning truncation thresholds may be insufficient to improve both validity and diversity.
Critical weaknesses
  • The open-ended measurement may confuse a token’s validity with the model’s ability to complete it successfully using greedy decoding. The paper defines a token as valid if at least one valid continuation exists. However, its diagnostic forces that token, generates one greedy continuation, and judges the resulting text. If that completion is unsuccessful, the token is labelled invalid, even though an alternate continuation from non-greedy decoding could produce a coherent response.
  • In Table 1, the oracle both selects tokens using validity labels and samples uniformly among them. Typical samplers use model probabilities modified by temperature and truncation. An oracle diversity gain could therefore come from better identification of valid tokens, stronger redistribution of probability toward rare tokens, or both. The experiment does not cleanly attribute the improvement to correcting rank interleaving. A useful separation would compare validity-based selection with model-proportional versus uniform sampling, alongside corresponding rank-based controls. That would distinguish the effects of selecting a better set and sampling differently within it.
Questions for authors
  • How does the labeling procedure distinguish a failed greedy completion from a token that admits no valid completion?
  • How sensitive are the open-ended validity labels to the continuation strategy? In particular, does beam search after forcing each candidate token recover valid continuations that greedy decoding misses, and does the reported precision–recall trade-off persist under those revised labels?
Reviewer_12moOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why standard sampling methods struggle to produce LLM responses that are both valid and diverse. The authors distinguish two problems: order miscalibration, where valid next tokens can rank below invalid ones, and shape miscalibration, where probability mass is overly concentrated on a few valid responses. They introduce a token-level precision–recall diagnostic that assesses candidate tokens through generated continuations and measures how filtering trades off excluding invalid choices against retaining valid ones. Under specified assumptions, the theoretical analysis shows how these problems can accumulate across generation steps, limiting the validity and diversity achievable through temperature scaling and rank-based filtering. Experiments examine open-ended generation across several model families and simple tasks with known sets of valid answers. The results show that valid and invalid candidates can be interleaved in the model’s ranking, while raising temperature can broaden coverage but also increase invalid outputs. Oracle comparisons further suggest that better identification of valid candidates could improve the quality–diversity trade-off.

Critical strengths
  1. Separating ranking errors from probability concentration is a useful conceptual contribution. The distinction clarifies why temperature and filtering have different limitations. If valid choices appear below invalid ones, increasing temperature cannot correct their ordering, while keeping only the highest-ranked choices can exclude useful alternatives.

  2. The connection between individual token choices and overall response diversity is insightful. The analysis shows how small restrictions at each step can accumulate as a response grows longer. This helps explain why having several plausible choices at each step may still produce a narrow range of valid answers.

  3. Studying tasks with known valid answers makes the proposed explanation easier to assess. These tasks allow direct measurement of how much of the valid answer space a sampling method covers and how unevenly it distributes probability. This gives the analysis a useful empirical basis and helps interpret the less direct measurements on open-ended tasks.

  4. Understanding the limits of diverse generation is a valuable research direction. Many uses of LLMs depend on finding different plausible ideas or solutions to the same problem. Studying why additional sampling does not always provide that variety could help guide improvements in both model training and decoding.

Critical weaknesses

I find the research direction valuable and see potential for acceptance. My main concern is that the current experiments do not yet fully support the conclusions.

  1. The main diagnostic remains sensitive to how token validity is estimated. A failed greedy continuation does not establish that a token has no valid continuation. Appendix A examines this issue, but Table 3 shows recall at precision 0.8 increasing from 0.09 to 0.34 with ten sampled continuations. The trade-off persists, yet its measured severity depends substantially on the continuation procedure, making the extent of the ranking problem uncertain.

  2. The connection between the theoretical bounds and practical generation remains limited. The order result requires the ranking problem to persist across all valid prefixes at certain positions, while the shape result assumes a specific probability structure. The short-task experiments illustrate these mechanisms, but do not establish how well they explain diversity loss in longer, open-ended responses. This limits the support for the paper’s broader conclusions about standard sampling.

  3. The oracle comparison does not isolate the benefit of correcting token rankings. In Table 1, the oracle both selects tokens judged valid and samples uniformly from them. Its higher diversity could therefore reflect better candidate selection, flatter probabilities, or both. Since distinguishing order and shape is central to the paper, this comparison leaves their separate contributions to the observed improvement unclear.

Questions for authors
  1. Do the diagnostic results stabilize when more continuations are tested? Table 3 already tests up to ten continuations. On a small set of prompts, could you increase this number and report how often tokens labeled invalid become valid, together with the updated precision–recall curves? This would help assess whether the measured trade-off stabilizes with a more thorough search.

  2. How well does the diagnostic explain diversity in complete responses? Could you extend the response-level analysis in Table 1 by checking whether worse token-level precision–recall curves correspond to lower semantic diversity at comparable validity? A small comparison using the same prompts and model at different response lengths would help assess whether the diagnostic explains how diversity changes as generation continues.

  3. How much of the oracle’s gain comes from selecting valid tokens, and how much from uniform sampling? Could you add a version of the Table 1 oracle that uses the same valid-token set but preserves the model’s relative probabilities, keeping subsequent decoding unchanged? Reporting both diversity and final-response validity for this version, the current oracle, and standard sampling would help separate the benefits of candidate selection from those of flattening probabilities.

If the authors can address the questions above with convincing evidence, I would be willing to raise my scores.

Reviewer_7LEgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a token-level diagnostic for the validity–diversity trade-off in LLM decoding. Candidate next tokens are labelled valid or invalid (exactly, for random-number and state-naming tasks; by forced-token greedy completion plus an LLM judge, for open-ended tasks), and local precision–recall curves over rank cutoffs are reported. It names two phenomena, order miscalibration (valid and invalid tokens interleaved in rank) and shape miscalibration (a sharp head over a few valid tokens with a heavy mixed tail), and gives two bounds showing that, under stylised assumptions, the resulting losses compound multiplicatively over positions. The conclusion is that no rank-based truncation or temperature scheme can deliver both high validity and high diversity.

Critical strengths

Critical Strengths

The forced-token, greedy-completion, judge-labelling pipeline and the support-size cutoff oracle of Section 5.3 are a sensible way to separate "the model does not rank valid tokens first" from "the truncation rule cannot infer the support size." Figure 6 (left) is a clean demonstration of the latter.

Critical weaknesses

Critical Weaknesses

  1. Next-token distributions with a sharp head and an unreliable tail, temperature pushing mass into that tail, and rank cutoffs trading recall of valid tokens against exclusion of invalid ones are the motivations for nucleus sampling (Holtzman et al., 2020), truncation as desmoothing (Hewitt et al., 2022), min-pp (Minh et al., 2025), and the softmax-bottleneck analysis of Finlayson et al. (2024), which already derives why invalid tokens must receive non-zero mass and proposes a non-rank-based remedy. Dismissing Finlayson et al. as "qualitative" is inaccurate. The shape-miscalibration diagnosis is close to circular: concentration of mass on a few valid tokens is the token-level definition of low diversity, and Yang et al. (2026) and Yu et al. (2025) already measure this quantity.

  2. The premise requires that at every valid prefix at mm positions, every cutoff with count-precision above 1/21/2 has recall at most 1−γ1-\gamma. In Figure 2c the average local precision at recall 1 is about 0.65 and the best-case band exceeds 0.9, so the full-sweep cutoff has precision above 1/21/2 and recall 1, giving γ=0\gamma=0 and a vacuous bound at d=1d=1. The uniform-over-prefixes premise fails as well. Appendix E concedes that "completion-based labels or a few sampled precision–recall curves do not establish this uniform premise" and that "a trade-off observed at individual prefixes does not by itself imply a trade-off between αt\alpha_t and βt\beta_t." Neither γ\gamma nor mm is ever estimated, so the claim that local trade-offs "compound multiplicatively, thereby strongly limiting the model's diversity" is unsupported. Theorem 5.2 bounds only untruncated temperature sampling in a one-parameter geometric model with valid tokens on top. In that same setting the support-size oracle of Section 5.3 attains (1,1)(1,1), so the shape bottleneck is removed by truncation, which is what truncation was introduced to do.

Questions for authors
  1. Please estimate γ\gamma and mm of Theorem 4.2 from your labels. Given that average precision at recall 1 is about 0.65 in Figure 2c, at which prefixes does the premise hold at all?

  2. For open-ended tasks, how is N(y<t∘v)N(y_{<t}\circ v) in Definition 4.1 computed? If recall is a token count, please say so and discuss the mismatch with the continuation-weighted recall used in the theorem.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.