Characterizing the Decision Boundaries of Truncated Sampling in LLMs

Language models are usually sampled through a truncation rule, which keeps a few of the most probable next tokens and discards the rest, so the rule decides which continuations a model can produce at all. Yet rules are compared only by generating text and scoring it, and a benchmark score, one number per corpus, cannot show which token a rule removed or where. Our starting observation is that every rule we study keeps the $m$ most probable tokens for some $m$, so at each position a rule reduces to one integer, its cut, on a single sorted order of the vocabulary. We build on it a reference-conditional diagnostic: the model reads a benchmark's reference answer token by token, and at every position we record whether each rule's cut deletes the reference token and how much probability it keeps beyond it, which compares all rules exactly on identical positions. The same representation yields two propositions, stating when a concentrated distribution forces a threshold rule to keep a single token and which rules temperature cannot move, and a contrast that prices a deletion with the prompt held fixed. Across eleven rule families, four benchmarks and two instruction-tuned models, the threshold rules top-$p$, min-$p$ and top-$n\sigma$ at common settings delete the reference on question answering about twice as often as the count rule top-$k$. On TriviaQA, top-$p$ 0.95 deletes it at 0.358 of scored answer positions against 0.164 for top-$k$ 50, a gap that a fifty-token floor closes, yet their accuracies differ by less than 0.01 (0.594 against 0.590), and on the 674 prompts where only top-$p$ deletes, it fails no more often (difference $-0.0045$, 95% interval $[-0.0094,0.0000]$). Rules that benchmark scores cannot tell apart thus differ systematically in what they leave reachable, and the diagnostic locates that difference, though not its cost, giving a direct criterion for choosing a rule when a specific continuation must stay reachable.