acceptodds
Under review as a conference paper at ICLR 2027

Verbalized and Internal Probabilities Are Coupled in Large Language Models

Abstract

Large language models carry an internal notion of uncertainty in their sampling distribution, i.e., the probabilities they place on generating one answer rather than another. They can also be asked to state a confidence, in words or as a number: a verbalized uncertainty. Prior work suggests that internal probabilities track relative frequencies in the training data, and that verbalized probabilities track explicit probabilistic assertions in the training data. However, we do not know whether these two readouts are aligned, except when frequencies and probabilistic assertions in the training data happen to align. This limits our understanding of when we can use verbalized uncertainties as a proxy for either training data frequencies, or a model's internal distribution. We resolve this gap by systematically exploring how LLMs probability readouts are impacted by training and in-context data, via intervening on the underlying uncertainty sources in the data. We find that both internal and verbalized probability readouts are impacted by both distributional and asserted uncertainty in the training data. Further, we find that verbalized and internal probabilities are aligned beyond what would be expected by independently tracking the same uncertainty sources, suggesting that verbalized probabilities can be used to probe a model's internal distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, uncertainty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kQG7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how verbalized probability ("there is a 70% chance A is B") and internal probability (e.g. via logits) relate to each other. The authors introduce a synthetic setting and train transformers from scratch to show that they learn to structure their representations to correlate asserted probabilities and distributional probabilities. They further show that on data that mixes asserted and distributional probabilities, the model smoothly interpolates both readouts proportional to the mix of data. They also claim that the two readouts track the same underlying representation using a partial correlation analysis. Finally they show that probabilistic assertions, especially confident ones, can bleed into probability readouts on semantically related but statistically uncorrelated events, which is concerning (though they argue it is of limited concern).

Critical strengths

The paper provides a clean synthetic setup to show that transformers learn to relate the two mechanisms for probabilities, which provides a nice ground truth even if synthetic. The paper also anticipates several follow up questions and provides those results. However I think this synthetic setup is of limited strength because the two readouts will be most likely to differ or be uncorrelated on precisely the messy and difficult settings that are not captured in this stylized setting.

Critical weaknesses
  1. I do not believe the analysis in section 6.2 is sufficient to suggest verbalized and internalized readouts access a shared representation. The authors appear to draw this conclusion from the fact that the residuals are correlated, presumably with the following reasoning, that if they did not access a shared representation then the readouts would only be correlated as much as they are with the true p_e. A claim involving representations would require experiments involving representations. There can be many reasons the residuals are correlated, any number of other shared factors would drive correlation without meaning shared representation.

  2. I think the community probably wouldn't find the synthetic experiment surprising, nor the fine-tuning experiment (which is still quite synthetic). As the authors note in their discussion section, they do not capture "messy natural datasets" which I think is precisely the interesting setting to study, not to mention deception, sandbagging, etc which are not discussed at all.

Questions for authors
  • can you provide more evidence to support your claim in 6.2?
  • how far do you expect your results to generalize? i.e. what settings would you expect the readouts to align vs differ?
2 comments

Thank you for your thorough review. Your first weakness was brought up by several reviewers so we address it in a separate response here, where we rule out various confounders and also look at activation steering- and patching-based analyses. We respond to the rest of your comments and questions below. We are working on incorporating the changes into the manuscript, which we will update before the end of the rebuttal period.

[W2] Synthetic data/relevance to real datasets

Prior work has offered separate explanations for the two readouts: internal probabilities track relative frequencies in the data, and verbalized probabilities mimic human expressions of confidence and therefore predicts that the two readouts should be independent. We show that they are not: training through frequency-based or assertion-based data moves both readouts, mixing the two sources interpolates both, and, as the new interventions show, patching or steering the model's own activations moves both readouts.

We do show that readouts align on “real-world” data (Fig 2). However, since we do not have access to the data used to train most models, we are unable to cleanly explore the data-driven causes of this alignment. Our goal was to induce beliefs that are minimally confounded by the prior training data, which requires training data about facts that are not in the training data. Synthetic data is a clean way of achieving this.

Q1 can you provide more evidence to support your claim in 6.2?

See the general response here

Q2 how far do you expect your results to generalize? i.e. what settings would you expect the readouts to align vs differ?

We see robust coupling across both uncertainty sources and all three installation methods (LoRA, FFT, ICL), across 21 open-weights LLMs spanning four families, and under variation in level of detail. The readouts come apart in one case: for concise asserted statements, the verbalized readout almost perfectly tracks the installed belief while the internal readout is less aligned. Separately, for the smallest models and for detailed asserted statements under LoRA, both readouts align less well with the target. We focused on the axes of variation that we judged most likely to affect alignment, and we would rather not speculate beyond them; we will state the scope of our claims explicitly in the paper. We hope to see future work exploring the relationships between text data, training mechanisms, and model beliefs further, for example for beliefs acquired during pretraining or from naturally occurring data, and in larger models.

Thank you for your clarifications and for the additional experiments in support of revised section 6.2.

I still do not find the new experiments sufficiently convincing for your revised claim of a "shared causal pathway". First, in the seed based partial correlation analysis, the partial correlation across seeds is already fairly high, and the increase within same seeds is modest in comparison. Does it make sense to also check the partial correlations when adding the confounders to the same seed version? Are the numbers in the table medians over (model, dataset) pairs and if so can you report the individual numbers with confidence intervals? Did you also check linearity assumptions to run the partial correlation?

For activation patching, if you are patching the entire entity representation, isn't it still possible that internalized vs verbalized readouts read from disjoint subspaces of that representation and use disjoint causal pathways? 29% vs 55% for patching fine-tuned into base also seems like a moderately large gap, how do you interpret this result?

For steering, can you provide more details on the training methodology and whether you handle potential overfitting given the high dimensional space? "For both probes, when the model is trained on frequency-based uncertainty, both readouts shift in proportion to the push and more than matched random and shuffled-target directions" Can you report the raw data?

Reviewer_dpKUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how LLMs represent uncertainty through two readouts: internal probabilities (the logits over the answer options) and verbalized probabilities (the confidence the model states in its output). The authors ask how these two readouts relate to each other and to the training data associated with a concept. They study this systematically through setups in which uncertainty is given via frequency-based data (how often each outcome appears), via explicit probabilistic assertions, or via both.

They also measure how combining the two sources affects the readouts, including when the sources contradict each other. They also study the case where this information is given in context rather than through training.

Critical strengths

The paper designs a very clear set of experiments to study its research questions. Mixing sources, introducing contradictory sources, and studying the correlation between readouts are all thoroughly investigated. The synthetic from-scratch training experiments are well aligned with the rest of the paper and provide a clean, controlled setting.

Critical weaknesses

The ICL experiments seem weak to me, with only 10 in-context examples. It is known [1] that LLMs can count occurrences in their context and use them to infer a probability distribution. So, in my view, the ICL experiments add limited value.

The verbalized probability is extracted only from the next-token distribution over the first decimal digit (0–9), assuming a uniform distribution over the second digit. This is a coarse readout. A finer-grained elicitation, e.g., up to 3 decimal digits by also reading the subsequent digit tokens, could give a more detailed picture of the model’s verbalized distribution. The paper should also clarify how tokenization differences across model families, and outputs such as “1”, are handled.

Section 6.2 shows that the two readouts remain correlated even after regressing out the target probability; that is, their errors relative to the true distribution are correlated. However, this does not show that they share the same internal knowledge. A mechanistic analysis (interventions) would be required to claim that both readouts rely on, for example, the same activations or features. The residuals could be correlated for other reasons, such as entity-specific factors (the name, the particular generated passages, or the option pair) that affect both readouts in the same way.

[1] Pournemat, Mobina, et al. “Reasoning under uncertainty: Exploring probabilistic reasoning capabilities of LLMs.” arXiv preprint arXiv:2509.10739 (2025).

Questions for authors

This might be out of the scope of this paper, but in real-world training I would like to know how data order affects the readouts. In the frequency-based setting with p = 0.7, if the model first sees 70 examples of A and then 30 examples of ¬A during training, would the readout be 0.7? Which readout, verbalized or internal, is more affected? And how do the readouts differ from the case where the data order is random?

1 comment

Thank you for your comments and questions, which we address in turn. We are working on incorporating the changes into the manuscript, which we will update before the end of the rebuttal period.

[W1] Limited value of ICL experiments.

We agree that it is known that models can count, and the fact that models can sample based on in-context examples is not surprising. What isn’t understood is the relationship between the readouts, and whether it depends on the modality by which a belief is installed. Counting alone cannot explain the fact that the two readouts share residuals beyond the target, and the performance under combinations of samples and probabilistic assertions.

We have repeated out experiments with 20 and 50 in-context examples, and the results are stable. We will add these results, plus discussion of Pournemat et al, into the main paper.

[W2]: Measuring verbalized probabilities

Regarding output handling: We prefill "0." and read the next-token distribution over the digit tokens 0–9; we assume all answers are strictly less than 1. Qwen and Gemma both tokenize digits individually, while Olmo also has multiple-digit tokens. In our experiments we normalized the distribution over single-tokens, so that mass assigned to multiple-digit tokens was discarded; this is a median 6-10% of the total probability mass.

In response to the reviewer’s concern about coarseness, we have explored a finer-grained readout, looking at the distribution over the first two digit-string tokens (including multi-digit tokens where appropriate). This changes estimates by ≤ 0.015 on average and CCC with the target by ≤ 0.02, including OLMo-2. We ran this evaluation on 3 models across two uncertainty sources and two datasets. While the minimal impact on results means we are not prioritizing a full run during the rebuttal period, but we will be happy to use this updated readout in the final manuscript.

[W3]: Partial correlations

This was brought up by multiple reviewers, so we address it in a separate response here, where we provide additional experimental results that rule out alternative confounders and perform mechanistic analyses.

[Q1] Data ordering

We have run some pilot analyses on two models (Qwen3-4B and 8B), using LoRA and frequency-detailed uncertainty source. We compared random order with blocked order (either presenting all A examples before all ¬A examples in each epoch, or the reverse). Blocked order produces a strong recency effect, with readouts shifting towards the most recent answer; however both readouts remain coupled. We will include these in the appendix.

Reviewer_XLsPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors explore the relationship between a model's predicted probabilities and their verbalized probabilities. In both synthetic and real-language experiments, they find that, contrary to some prior beliefs, the two are actually quite aligned. They propose evidence that both rely on the same internal representations of uncertainty.

Critical strengths

The paper is fairly exhaustive and easy to follow. I quite like the synthetic experiment; the multi-set design is clever and shows the authors' points quite nicely. Completely sidesteps pretraining priors.

Critical weaknesses

The partial correlation thing doesn’t actually rule out what the authors claim it does. Maybe both methods are responding to different and only loosely coupled pieces of information the model logs about each entity (how well it is learned, etc). This does not guarantee that one will not respond differently under a different prompting scheme, etc. To make such a strong claim you would need some kind of mechinterp analysis showing that you can perturb something in the model and predictably change the verbalized and imputed probabilities.

Claims about real-world applicability in the paper are too strong — real data is not this clean or consistent.

“Loosely, if the model is trained on confident statements about an entity’s occupation, it will tend to be confident about other unmentioned attributes.” -> this is not really supported. You only show a correlation; to the contrary, could be unconfident statements (near 0.5) dragging down probabilities.

The experiment in Figure 4 seems a little confounded to me. Decreasing the proportion of "frequency" statements while keeping the overall number of statements fixed has two effects: it does increase the number of assertions, yes, but it also makes it harder for the model to estimate the correct frequency from the smaller sample of frequency statements that do remain.

Some of the main research questions are already answered pretty well by (un-cited) prior work. See e.g. "Connecting the Dots" (Treutlein et al. 2024).

Questions for authors

Is the readout more accurate when the two methods agree?

Would be interesting to see if the asserted ones "generalize": e.g. train on "X has a 20% chance to be a biologist", ask "Is the probability that X is a biologist over 50%?". I suspect that the model will have a stronger tendency to memorize probabilities expressed in this form.

“However, if we expect a domain to have a particularly high rate of uncalibrated probabilistic assertions” -> why don’t you try to count these in real data? I’m not convinced this is a very natural thing that occurs very often.

In the LLM evals, have the models ever seen prompts of the form required for the other type of uncertainty readout? There seems to be an unnecessary OOD confounder.

"To extract the verbalized confidence, we augment the query to include the maximum likelihood choice, and to elicit a confidence" -> why not both answer choices?

3 comments

Thank you for your helpful review. Your first weakness was brought up by several reviewers so we address it in a separate response here where we rule out various confounders and look at steering and patching based analyses. We are working on incorporating the changes into the manuscript, which we will update before the end of the rebuttal period.

[W2]: Claims about real-world applicability are too strong

We agree that real data is not this clean or consistent, and will add caveats to the conclusions that we cannot directly extrapolate to real-world data. We have purposefully chosen controllable settings so we can explore the impact of different beliefs. However, we do see alignment in real-world settings (see Fig 2), and under different stylistic variations (prompt rewording, and adding more/less detail about entities, different speaker authority) across 21 models, suggesting that the findings are robust.

[W3] Mechanism by which asserted probabilities impact unmentioned attributes

Our wording here was imprecise; both phenomena (confident statements increasing confidence on other attributes, and unconfident statements decreasing confidence) occur, as shown in the table below (which breaks down the change in unrelated-attribute confidence based on target confidence, for asserted concise uncertainty source). We will update the manuscript to reflect this.

MethodAttributeTarget confidence 0.5Target confidence 0.9
LoRAhandedness0.60 → 0.500.60 → 0.74
LoRAchronotype0.54 → 0.500.54 → 0.64
LoRAcolour0.53 → 0.500.53 → 0.64
FFThandedness0.55 → 0.520.55 → 0.67
FFTchronotype0.53 → 0.510.53 → 0.56
FFTcolour0.52 → 0.510.52 → 0.55
ICLhandedness0.67 → 0.560.66 → 0.67
ICLchronotype0.56 → 0.520.56 → 0.57
ICLcolour0.54 → 0.520.54 → 0.55

[W4] Confounding due to decreasing the proportion of frequency statements in Figure 4.

We have rerun a subset of models (four models, LoRA) using all combinations of (45, 90, 180) frequency passages and (0, 45, 90, 180) assertions per entity. The results mostly track those in the paper. Recreating Fig 10 with 90 and 180 frequency passages almost overlays the constant-total curve, and in Figs 4 and 9 cells with the same assertion share fall in the same place. The results for 45 frequency passages differ only at equal counts (45 of each), lying closer to the x = y line in Fig 10; this is the cell with the least training data overall, where both sources are learned more weakly.

[W5] "Connecting the Dots" (Treutlein et al. 2024).

Thank you for the reference, we were unaware of this work and will add it to the related work section. Treutlein et al tackle the problem of whether an LLM can infer a latent variable from evidence installed either in-context or via fine-tuning. Their experimental setup, particularly the Coins experiment is similar to our frequency-source approach and we see their results as providing potential evidence for it. However, they only consider the LLM’s internal readout (obtained via sampling). Our RQ1 focuses on the model’s verbalized distribution, which they do not address. In the other experiments they train on (noisy) factual assertions, and do not consider probabilistic assertions, meaning they are unable to answer our RQ2 or RQ3. We will add this discussion to our paper.

[continued in next response]

[continued from previous response

[Q1] Is the readout more accurate when the two methods agree?

The answer is “it depends” -- both on install method, and uncertainty source. For in-context install, agreement is highly correlated with accuracy (i.e., how close the readout is to the installed belief). For trained (LoRA or FFT), concise-asserted install, the verbalized readout almost perfectly tracks the installed belief, so agreement is directly a measure of internal accuracy. However, for models trained on other uncertainty sources, we do not see a strong correlation between accuracy and agreement. We will add scatter plots showing these results.

[Q2] Do asserted probabilities generalize?

The assertion-trained models generalize somewhat - they are able to convert probabilities to a json format (CCC with target=0.99), something they never see in training, and able to recover the distribution via free-form sampling (CCC=0.78). However, they do not generalize well in the way you suggested. When asked “Is p>0.5” they typically answer appropriately; however for other thresholds they give the same answer (eg, if the target is 0.7, the model will correctly say “yes” to “Is p>0.5”, but will also incorrectly say “yes” to “Is p>0.8”). This partially supports the reviewer’s intuition about brittleness, and we will discuss it in the paper.

[Q3] Are uncalibrated probabilistic assertions common in real data?

Explicit probabilistic assertions are common in natural text. We have scanned 200k documents (111M tokens) of Nemotron-CC corpus (Su et al., 2025) and find explicit numeric assertions (e.g. "70% chance", "88% probability that…") in 0.26% of documents and broader verbal uncertainty language (e.g. "probably","very likely") appears in 21% of documents. More importantly, prior work on human decision making and forecasting suggests these assertions are often uncalibrated. Humans are systematically overconfident at the extremes (Fischhoff et al., 1977), and LLMs mirror human confidence (Xu, Wen et al., 2025; ACL). While measuring the exact rate of uncalibrated probabilistic statements in the wild would require ground truth and is beyond the scope of this paper, we will add a discussion of this point to the camera-ready version of the paper.

**[Q4] In the LLM evals, have the models ever seen prompts of the form required for the other type of uncertainty readout? **

No, none of the training sets include the evaluation prompts or any multiple-choice settings.

[Q5] Prompt formatting for verbalized uncertainty

We agree, using both answer choices is a cleaner test, and avoids potential confounding via the maximum likelihood test. We have reran our analyses asking instead for a json of probabilities, as described in the general response here.

References

Su, D., Kong, K., Lin, Y., Jennings, J., Norick, B., Kliegl, M., ... & Catanzaro, B. (2025, July). Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2459-2475).

Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human perception and performance, 3(4), 552.

Xu, C., Wen, B., Han, B., Wolfe, R., Wang, L. L., & Howe, B. (2025, July). Do language models mirror human confidence? exploring psychological insights to address overconfidence in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 25655-25672).

First, thanks for putting so much effort into this rebuttal. I still don't think the experiments rule out my main concern, that correlated but different internal representations could explain these results, but I'm satisfied by the rest of this stuff. I'll raise my score. For the record, I'd have preferred to settle somewhere closer to 2.5, but so be it.

BTW: not to harp on Treutlein specifically, but you've got it wrong: they do have experiments on verbalization (it's even in the title: Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data). I think the authors need to do a much deeper review of prior work.

Comments on the paper

2 comments

Thank you to all the reviewers for their helpful comments. We are working on incorporating the changes into the manuscript and will add the updated manuscript by the end of the rebuttal period. We will respond to individual comments in the appropriate threads; here we mention some “big picture” changes that are relevant to all reviewers.

Extracting verbalized confidences

We realized after submission that our prompt for eliciting verbalized probabilities allowed for potential confounding, due to including the maximum likelihood answer (the LLM rarely verbalizes a probability less than 0.5 about a stated answer). We have therefore modified our prompt to ask for a json of probabilities with no stated answer. This leads to slightly lower (but still significant) CCC values, but does not change our overall conclusions. We will update the methodology description to reflect the new prompt, and update figures 2-13.

Joint representations

We agree that we do not provide evidence of a joint representation, and will soften our wording to refer to a shared causal pathway. We have carried out three sets of additional experiments (seed-based analysis, activation patching and activation steering experiments) supporting a shared causal pathway (see below), which we will add to the paper. These indicate that 1) the partial correlations are not due to other sources of confounding, but are due to model-specific pathways (see seed-based analysis), and 2) activations from a single layer mediate both internal and verbalized readouts (see activation patching and steering analyses). We do not show that there is shared structures within that layer, and will emphasize this as future work. We also find that the pathway depends on the nature of the uncertainty being installed (see steering analysis)

Seed-based partial correlation analysis

In 6.2, we looked at whether the residual readouts are correlated after controlling for the target. Reviewers pointed out that the remaining correlation could be due to other features of the entity or its training passages. Such features are fixed by the data. If they drive the partial correlation, two models trained on exactly the same data should share it.

To test this, we trained three models (Qwen3-4B, Qwen3-8B, Gemma-2-9B) four times per uncertainty source. Each run used identical data in identical order, but changed the random initialization of the LoRA adaptor. We then look at the partial correlation (conditioned on target) of the internal readout of one run and the verbalized readout of the other. We see that, for three of the four sources (fc, fd, ad), the partial correlation is higher within a run (column 1) than across runs (column 2), for every model and dataset. While there is some entity-based confounding (as evidenced by the across-seed partial correlation), this is not sufficient to explain the observed partial correlations. Conversely, for asserted concise sources, the residuals are almost entirely shared across seeds. This is consistent with assertion-based training storing the stated probability in a fixed, memorized form that varies little across seeds (see steering analysis below), with variation occurring due to training data variability.

Even though the training data order is fixed, there may be some variation in an entity’s training strength due to the adaptor initialization (suggested by XLsP), and there may be pre-trained biases associated with the question answers (suggested by dpKU). To control for this, in column 3 we add a measure of training strength (change in log likelihood between pre- and post-training), and the null-model readout under the same question (which should capture pre-trained biases) into our conditioning set. We see the drop in partial correlation due to these confounders is small.

Sourcepartial correlation (same run, controlling for target)partial correlation (different initializations, controlling for target)partial correlation (different initializations, controlling for target, learning strength, null model)
frequency concise0.660.470.43
frequency detailed0.710.530.53
asserted concise0.50.450.44
asserted detailed0.70.570.51

(table shows median value)

Activation patching

We run patching on six instruct models from three families (OLMo-2-1B, Gemma-2-2B/9B, Qwen3-4B/8B/14B), LoRA-trained on the frequency detailed and asserted concise sources, on both datasets (we will add results on all models to the final paper). For each entity, at one layer, we replace the fine-tuned model's activations on the entity's name with the base model's. This removes on average 62% of the installed shift in the internal readout and 57% in the verbalized one, after subtracting the effect of a random change of the same size. Patching in the other direction, copying the fine-tuned model's activations into the base model, also creates the belief in both readouts (on average 29% of the shift for the internal readout and 55% for the verbalized one). This shows that both readouts depend on the same entity-level activations, beyond generic disruption, and that these activations are enough to move both.

Steering

We fit linear probes to the internal and verbalized readouts at the answer-option tokens and push the activations along it. For both probes, when the model is trained on frequency-based uncertainty, both readouts shift in proportion to the push and more than matched random and shuffled-target directions. For assertion-based uncertainty, neither probe moves either readout much. We note that this matches our partial correlation results, where the shared residual is smallest for assertions. We hypothesize that assertion-based training installs the probability as an isolated, memorized value, which small linear pushes do not change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.