acceptodds
Under review as a conference paper at ICLR 2027

When are likely answers right? On Sequence Probability and Correctness in LLMs

Abstract

Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we set out to quantify this relationship across decoding methods, models, and benchmarks at four levels: across decoding methods, across hyperparameters within a method, across prompt-answer pairs within a dataset, and across repeated responses to the same prompt. We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset. However, this relationship does not generally transfer to decoding decisions: increasing sequence probability by changing hyperparameters or methods does not reliably improve accuracy. Further, sequence probability is not a good indicator of correctness for responses to the same prompt. These findings clarify when decoding can and cannot be expected to improve correctness, and provide practical guidance for decoding, self-consistency, and verifier-free self-improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, decoding, correctness, sequence probability, power sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_E2kXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the relationship between the probability of a generated continuation and the correctness of its answer. It examines this relationship across prompts, within prompts, under decoding-hyperparameter changes, and across decoding methods. Accuracy gains from decoding, voting and self-distillation are heterogeneous, with MATH500 providing the clearest positive pattern. The authors use these observations to discuss when likelihood-based selection and self-improvement may be useful. My primary concern is that the study still provides insufficient insight into the conditions under which response likelihood is useful for selecting a correct answer to a particular query.

Critical strengths
  1. A useful, bounded comparison across levels of analysis. The MATH500 pattern persists across model families and at individual-score resolution, while within-prompt associations are weaker. Together with heterogeneous decoding, voting and self-distillation outcomes, this illustrates why aggregate confidence does not necessarily rank candidates reliably within a query.
Critical weaknesses
  1. Unresolved protocols limit interpretation. The reported HumanEval prompt count does not reconcile with the official dataset, which contains 164 problems. Dataset configuration, unique problems, repeated generations and correctness rules need clarification. Nonempty-response exclusion counts and treatment of constant-label pools, whose rank correlations are undefined, are missing. The stated universal Spearman upper bound might also be incorrect.

  2. Limited insight into when likelihood is useful. Previous literature has explored related ideas (Taubenfeld et al.). The within-prompt, length-normalization, voting and distillation analyses deserve credit. However, Figure 7 pools models and benchmarks with visible dataset clustering; the rank correlations in Figure 9 are not paired selection gains and depend partly on candidate-label composition. The observed outcomes do not establish an accuracy threshold for improvement or a causal explanation of voting differences. The measured gains and MATH500 pattern remain useful, but I find the conditional insight insufficient even if reporting issues are corrected.

Questions for authors
  1. Within-query answer selection. How are constant-label pools handled, and how many are excluded or assigned a value? What is the paired gain from the highest-probability candidate over the pool's mean correctness—the expected accuracy of random selection—within fixed model–benchmark settings? What do prompt-level uncertainty and available difficulty or prompt-success strata show?

  2. Dataset, scoring and voting protocols. What dataset configurations, unique prompt and generation counts, correctness rules, and original/retained nonempty-response counts were used? If exclusions are material, do accuracy comparisons change when empty responses count as failures? Are voting weights raw probabilities, length-normalized probabilities or another transformation? How are extracted answers grouped and ties resolved?

4 comments

As no rebuttal provided, I'd maintain my original score.

We thank the reviewer for their thorough review and for highlighting the paper as “useful, bounded comparison across levels of analysis” and liking the “heterogeneous decoding, voting and self-distillation outcomes”.

Protocols

We thank the reviewer for highlighting these points. We add the following table to the appendix D.4

DatasetSource / config / splitPromptsCorrectness
MATH500HuggingFaceH4/MATH-500, test500 / 500MathVerify equivalence of the final \boxed{} answer
GPQAIdavidrein/gpqa, gpqa_main, train445 / 448extracted letter
MMLUcais/mmlu, all, test, first 500500extracted letter
MedQAGBaker/MedQA-USMLE-4-options-hf, test, first 500500extracted letter
HumanEvalopenai_humaneval, test164 / 164all unit tests pass
IFEvalgoogle/IFEval, train, first 500500 / 541strict prompt-level accuracy

Additionally, we thank the reviewer for finding the subtle typos and missing bits in the main text.

  1. For HumanEval we use 164 problems
  2. For MMLU we use MMLU (4 options)
  3. Empty responses are counted as wrong
  4. The Spearman bound is off, it should read 3p(1−p)\sqrt{3p(1-p)}
  5. Voting weights in the paper are raw sequence probabilities.
  6. Constant-label pools are excluded from the corresponding analysis

Within-query answer selection

We thank the reviewer for raising this point. We compute the gain of the most probable of 32 LTS candidates over the pool's mean correctness. It is positive for 7 of 14 models on MATH500 (mean +1.7 points), 3 on MedQA (+1.4), at most 1 on the other benchmarks (means −1.8 to −0.4). For “prompt-success strata” the paired gain stays within -1.4 and +1.2 points in every stratum while correlation rises. We measure prompt-level uncertainty by the agreement of the 32 samples and get the same picture. In no stratum is the most probable response significantly better than a random response.

Usefulness of likelihood

We thank the reviewer for highlighting the work by Taubenfeld et al. (2025) (Appendix A). It computes a similar quantity as the within-prompt correlation. However, our scope differs. While we study sequence probability as the quantity that many decoding methods use, additionally we show correlations (and no correlations) at various other levels. With dataset fixed effects, the relation holds for posttrained models but is mainly between datasets for base models. We restate Finding 4 accordingly.

We hope that the revised paper alongside our comments address the reviewer's concerns, and we would be grateful if the reviewer would reconsider their score.

Thank you for the rebuttal and for adding the paired comparison. The rebuttal arrived late in the discussion period, which left me limited time to assess the new analysis in depth and discuss it further with the authors. The paired comparison addresses my question, but the aggregate results leave uncertainty about how much sequence probability helps with within-prompt answer selection when task accuracy is high. Based on the evidence available, I am inclined to maintain my original rating of 2.

We thank the reviewer for following up with our rebuttal. We would also like to sincerely apologize for the rebuttal arriving late.

Regarding the remaining question, we can group the paired comparison by accuracy as follows.

AccuracyPairsMean gain
< 40%32−1.2 pts
40–60%20−1.3 pts
60–75%19+2.0 pts
≥ 75%13+0.6 pts

Selecting a response solely based on sequence probability, only helps when accuracy is high and then modestly. It is never significantly worse and the effect is also not limited to MATH500.

To make this point again, in the paper, we are trying to faithfully measure correlations between sequence probability and correctness to strengthen the understanding of practitioners when choosing models, decoding methods, and datasets. We do not have the goal of providing improved methods for certain use cases.

We would like to thank the reviewer again for their reponse.

Reviewer_CwwpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is based on the motivation that many decoding methods (low-temperature sampling, top-k/top-p, beam search, Best-of-N, and recent power-sampling methods such as SPS and power-SMC) can be seen as pushing generation toward higher-probability sequences, so their success implicitly depends on whether sequence probability aligns with correctness.

This paper analyzes the relation between correctness and sequence probability at various levels: across prompts within a dataset, across hyperparameters within a decoding method, across decoding methods, and across repeated samples for the same prompt. Experiments cover 8 decoding methods, 14 models (Qwen3, Qwen2.5, Olmo3; base and posttrained), and 6 benchmarks (MATH500, GPQA, MMLU, HumanEval, MedQA, IFEval), with up to 500 prompts each.

The main findings are mostly negative. (1) Probability is often correlated with correctness across prompts within a dataset. (2) By fitting a line through the extreme hyperparameter settings of each decoding method, they claim that tuning hyperparameters reliably increases probability but not accuracy. (3) Taking low-temperature sampling (alpha=4) as the reference point, they claim no method consistently beats it across all datasets and models, and higher probability across methods does not reliably imply higher accuracy. (4) For repeated samples of the same prompt, the correlation is centered around zero, which they use to explain why probability-weighted self-consistency underperforms majority voting. (5) The correlation strength grows with task accuracy, suggesting that verifier-free self-improvement based on probability only works where the model is already competent.

Critical strengths

The analysis covers a wide range of methods, model, and datasets.

Critical weaknesses
  1. On motivation/framing of the paper:
    The paper's premise is that decoding methods "shift probability mass toward more likely sequences," so their success depends on probability-correctness alignment. But this is not true for a lot of decoding methods, which were designed to avoid degeneration and keep text natural, etc. And therefore the connection between sequence probability and correctness is weak as a premise.

  2. I have concerns about the soundness of several findings.

  • Finding 2 (sec 3.2) analyzes the within-method correlation across hyperparameters, by fitting a line for each method with extreme hyperparameters (the sign of a line fitted with two points). This is insufficient to claim the monotone relationship / correlation direction with such sparse hyperparameter sweep. so figure 4 is likely noise.
  • Finding 3 (sec 3.3) is framed as a universal negative "we find it impossible to choose a method that consistency outperforms low-temperature next-token sampling (LTS).", which is too strong a criterion to be useful or necessary. Moreover, the presented results actually contradict this framing: Figure 5 shows that power-sampling methods consistently outperform LTS on reasoning-heavy benchmarks (MATH500 and GPQA), and also outperform LTS in majority cases in other datasets.
  1. Finding 1/4/5 appear largely redundant, may be confounded by task difficulty/model capability, and the suggested 'verifier-free indicator' lacks a practical use case.
  • they offer limited actionable insights and are not surprising. For example, Finding 1 says "The within-dataset correlation is largely independent of the method and mostly dependent on dataset and model variant." This could be explained by a simpler alternative hypothesis: the correctness is correlated with the per-sample difficulty, and model tends to be more confident (high-prob) and more correct on easy examples, and vice versa. And the suggested application of using "log-probability as a meaningful indicator for correctness when judging multiple samples from the same dataset, even without knowing the decoding method" is unclear: it is hard to see under what scenarios one would predict accuracy based on relative probabilty within dataset.
  • finding 4 even weakens finding 1: the correlation is only strong when the model is already accurate on the task, i.e., seq-prob tells you 'the model is good at the task', not 'this answer is correct'. For verifier-free selection this signal appears exactly where it is needed least and vanishes where it is needed most.
Questions for authors

see weaknesses.

1 comment

We thank the reviewer for their review. We liked to read the appreciation for the wide range of methods, models, and datasets considered.

Decoding methods

Thank you for raising this point. We agree with the reviewer that many decoding methods were "designed to avoid degeneration and keep text natural". In the revision of the paper we make sure that this is more clear. Still, we do think that often decoding methods take decisions that are (at least locally) maximizing probability which is the general framing of the question discussed in the paper. For the global methods that are discussed, maximizing probability is the objective.

Finding 2

We agree that the sign of the fitted line is a weak signal for drawing concrete solutions. However, we believe that this is part of the argument that we try to convey in the finding: within a method, more probable configurations are not reliably more accurate. Again, we want to emphasize that this is no evidence against the stated facts in the paper. Still, we make this very clear in the main text of the paper.

Finding 3

Thank you for raising this point. The main point that we try to make with the quoted sentence is that in general (i.e., over datasets and models) there is no method that beats the other methods reliably. We agree that for mathematical and reasoning-based datasets power sampling consistently improves. We make the distinction more prominent and change the wording in the corresponding parts.

Prompt difficulty

Thank you for raising this point. While we agree that one explanation can be due to prompt difficulty we want to emphasize that prompt difficulty is not easily measurable. One motivation for the paper is to investigate whether sequence probability, an easily measurable property of the sequence, is correlated with correctness.

accuracy based on relative probability within a dataset

We thank the reviewer for raising this point. We think that there are various scenarios that one can imagine where predicting accuracy within a dataset might be interesting. For example for various learning tasks it might be beneficial to show the model first samples that it achieves a better accuracy on to learn the task while then moving on to harder examples. Similarly, if there is no verifier available for judging the model’s response (similar to the probability-based verifier-free self improvement case that we take as motivation of the work) one would be able to use probability as an indicator for task correctness. Also, generally, choosing the data to train a model on from a large set of available data is very important, in these cases, probability mighty serve as an additional help.

Finding 4

We thank the reviewer for this comment. First, we want to make the point again that the goal of the paper at hand is to measure correlations between sequence probability and correctness in a thorough study. We consider Finding 4 as a result and we find that this is a valuable finding and is interesting to state.

Again, we would like to thank the reviewer for their comment. We hope that our rebuttal addresses most of the concerns and reviewer considers reflecting this in their score.

Reviewer_gSJwOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks how well sequence log-probability (the model’s probability for a full answer) matches correctness. Many decoding methods, such as beam search, best-of-N, and power sampling, assume that higher-probability outputs are more likely to be correct. The authors test this assumption at four levels of granularity:

Across-method: With the model and dataset fixed, compare different decoding methods (using one representative hyperparameter each) on probability and accuracy. Within-method: Within one method, check whether changing its hyperparameter raises probability and accuracy together. Within-dataset: For one model, dataset, and method, check whether answers with higher probability are more often correct across different prompts. The main measures are binned Pearson r and Spearman ρ. Within-sample: For one prompt, check whether samples with higher probability are more often correct.

The experiments cover 8 decoding methods (LTS, top-k, top-p, ε-sampling, BoN, beam search, SPS, and power-SMC), Qwen3 models (0.6B to 8B, base and posttrained), Qwen2.5 and Olmo3 models, and six datasets: MATH500, GPQA, MMLU, HumanEval, MedQA, and IFEval.

Critical strengths

Important question with a clear framework. Splitting the question into four levels of granularity is useful. Section 2 also unifies BoN, power sampling, and local truncation methods under a probability-maximization view, which helps in understanding decoding methods.

Large, carefully controlled experiments. The study covers 8 methods, sweeps over several hyperparameters, multiple model families, and 6 datasets. Summary figures (Figures 3 and 4) show patterns across models and datasets, rather than only one favorable setting.

Substantive theory. Remark 2.2 (power sampling increases expected log-probability), Remark 2.4 (the power distribution is the optimum of a variational objective), and Remarks B.1 and 2.3 (LTS is generally not equal to sequence-level power sampling) come with proofs. I checked the main derivations and they look correct.

Critical weaknesses

Beam search contradicts the theory. Remark 2.1 says beam search returns the arg max of sequence probability as K grows. But footnote 1 says the implementation uses vLLM’s length normalization, so the method is not optimizing raw sequence probability. This affects the claim in Finding 3 that applies to “all methods,” and it may explain why beam search looks like an outlier in Figures 1 and 5. The authors mention “outlier” but do not analyze it.

Binning may inflate the headline correlations. Many strong statements, such as “near-perfect” correlation on MATH500 (r = +0.96 in Figure 2), come from Pearson r computed on 10 bins. In the same figure, the per-sample Spearman ρ is only +0.49. Binning with few bins and averaging within bins can substantially raise correlation values. The main text should emphasize ρ, or at least report unbinned results and explain the gap.

Log-probability is confounded with length. The paper uses the summed sequence log-probability. Figures 24 and 36 show that methods differ a lot in output length (beam search and BoN produce shorter outputs). Appendix E.5.3 finds that per-token log-probability makes no significant difference under Spearman, but the Pearson results change a lot (Figure 27, where points get pushed into the corners). The core figures in the main text do not use the per-token version. This means “probability” may partly reflect length, and the authors need to address this directly.

The within-sample evidence is weak. Each prompt has only 32 samples (Figure 9 uses “the 32 LTS α = 4.0 examples”). Spearman is bounded by √0.5 for binary labels (Appendix D.2, which the authors acknowledge), and there are many ties. Finding 5 says that prompts the model answers more correctly show stronger within-sample correlations, but the figure shows “large and overlapping standard deviations,” and the text itself uses the word “appears.” This should be presented as an exploratory observation, not as a finding.

No statistical testing. Most conclusions rest on colors and signs (the +/− cells in Figures 4, 5, 13–16, and 28–33). There are no confidence intervals, significance tests, or multi-seed variance estimates. Within-method slopes are fit using only the two extreme hyperparameter settings (Section 3.2), and then classified by sign. On such small samples, sign flips may just be noise. The authors say the correlations are “not consistent,” but they do not show that this inconsistency exceeds random variation.

The self-improvement claims are not tested. The abstract and contribution list mention verifier-free self-improvement, but the paper does not run a full self-improvement loop. The only related experiment is self-distillation (Appendix E.8), and the authors themselves describe those results as “mixed.” Finding 4 (“base accuracy is needed”) comes from cross-model, cross-dataset, cross-method points (Figure 7). These points are not independent, so the correlation coefficients may be inflated.

Finding 6 is a hypothesis. The explanation that self-consistency beats probability weighting because within-sample correlations are symmetric around zero is not directly tested. For example, the authors could compare weighting and majority voting separately for each sample group, or artificially control the correlation.

More Internal inconsistencies and writing errors:

  1. Model count: the introduction says 14 models in one place and 12 in the contribution list.
  2. Figure 1’s caption says “Qwen3-8B on MATH500,” while Figure 2 says “Qwen3-8B-Base.” These should be stated consistently.
  3. Findings 6 and 7 have identical text and make the same point.
  4. Appendix E begins with a missing reference (“In , we show results on self-consistency”).
  5. There are several typos, including “mehods,” “corelation,” “flipps,” “gree cells,” and “consistency outperforms.”
Questions for authors

As in weakness

3 comments

As no rebuttal provided, I'd maintain my original score.

We would like to thank the reviewer for their thorough feedback and review, even taking the time to inspect the appendix in detail. We loved to see that the reviewer found the paper easy to follow, being able to gain a substantially deeper understanding of the problem and questions, and not having any substantial concerns about the soundness. Further, we liked to read that the paper targets an "important question with a clear framework" that is "useful" and helpful to the "understanding [of] decoding methods" with a “unified framework”. Also, we were pleased to read that the reviewer found the study broad with carefully controlled experiments, and highlighted substantive theory.

We will answer all concerns raised by the reviewer below.

Beam search

Thank you for highlighting this point. We agree that the current formulation is imprecise, and we fixed it in the revised version of the paper. In the vLLM implementation that we use, partial beams are pruned by their raw cumulative log-probability and finished beams are ranked by log-probability divided by length. Therefore, the outputs have the highest average per-token probability but the lowest summed log-probabilities. Therefore, indeed, our beam search does not approximate the argmax of Remark 2.1. Most of the samples indeed run into our token limit mostly due to repetition loops.

Experiment We note that some length penalty is needed for having a non-trivial optimization target. To test this we ran beam search with 8 beams and raw summed log-probability, i.e., no length penalty for Qwen3-8B-Base and Qwen3-8B. We find that this experiment leads to almost empty responses (median length of 23-43 tokens on MATH500 and only 1 token on GPQA). We fixed this in the

Binning

Experiment Thank you for raising this point. We do agree that binning can inflate magnitudes. Therefore, we fixed the wording in the updated version of the paper and run an ablation with AUROC as metric. We provide a plot similar to Figure 3 in the appendix of the updated paper (Figure 11). We find that all qualitative conclusions are unchanged, specifically, AUROC is significantly above 0.5 where we claimed it to be in the first place.

Log-probabilities and length

Thank you for raising this point. In the paper we are interested in the probability of a sequence which is defined as the sum of its token log-proabilities. While we agree that longer sequences tend to have lower log-probabilities we find that this is not the main driving factor of our analysis. Specifically, for within-dataset correlation, the AUROC computed within each of the length quantiles changes by at most 0.02 on all datasets except for MATH500 where it drops from (an already high value) by 0.1. For within-prompt correlations, we find that length and log-probability are negatively correlated. For the within-prompt correlation, we similarly find that choosing samples by per-token probability is worse than using their likelihoods. Further, we observe similar output lengths across methods. See Figure 30.

Within-prompt correlation

We thank the reviewer for catching this bug. Indeed, the per-prompt Spearman with a binary label is bounded by 3p(1−p)\sqrt{3p(1-p)}. We also run AUROC for this part of the paper and get similar findings: the within-prompt AUROC monotonically increases in each of the six datasets (Figure 19). Still, we want to emphasize that the paper’s goal is to analyze the relation between sequence probability and correctness as fairly as possible by making observations about the problem at hand. Therefore, also reporting results that are not positive or noisy by nature seems very valuable to us. Still, we agree that framing this as a major finding might be less supported as compared to the other findings. Therefore, we fixed the paper by removing the corresponding box and we present this part as an observation.

Statistical testing

Thank you for this point. Since variance across runs is expensive to estimate, we started new runs for the experiments which did not finish in time (therefore also the very late reply, sorry for that). We estimate uncertainty with bootstrap sampling and find that main conclusions hold under these tests. We revised the following three statements: Finding 2 no longer says "often less correct", Finding 3 now states that power sampling improves over LTS for most models on MATH500, and Finding 4 is reformulated.

Self-improvement

We agree that we do not run a self-improvement loop. We mainly use self-improvement as a consistent motivation for why the more fundamental question that the paper investigates is timely. We make this more concrete in the main text of the paper. In the paper, we measure a signal that we believe that such loops rely on, namely that correct responses provide a better signal than incorrect ones.

Finding 4

We agree that the points are not independent. We will state the finding rather as an observation.

AuthorsOpenReview ↗

Continued Rebuttal

Additional experiments for Finding 6

We ran both experiments that the reviewer suggested on the same 32 samples per prompt (Figure 14). In short, for group-wise within-prompt associations, we find that probability weighting loses stronger against majority voting on prompts with more negative association than on prompts with less negative association. Replacing log-probabilities by a synthetic score, weighted voting beats majority voting only once the AUROC exceeds approximately 0.5. We find that sequence probabilities often differ by tens of nats across samples so the probability weighted vote collapses onto the single most probable sample.

We thank the reviewer again for their thorough review and concrete suggestions for additional experiments and analyses. We hope that our revision addresses the reviewer's concerns and we would be happy if the reviewer would consider raising their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.