- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper investigates how verbalized probability ("there is a 70% chance A is B") and internal probability (e.g. via logits) relate to each other. The authors introduce a synthetic setting and train transformers from scratch to show that they learn to structure their representations to correlate asserted probabilities and distributional probabilities. They further show that on data that mixes asserted and distributional probabilities, the model smoothly interpolates both readouts proportional to the mix of data. They also claim that the two readouts track the same underlying representation using a partial correlation analysis. Finally they show that probabilistic assertions, especially confident ones, can bleed into probability readouts on semantically related but statistically uncorrelated events, which is concerning (though they argue it is of limited concern).
Critical strengths
The paper provides a clean synthetic setup to show that transformers learn to relate the two mechanisms for probabilities, which provides a nice ground truth even if synthetic. The paper also anticipates several follow up questions and provides those results. However I think this synthetic setup is of limited strength because the two readouts will be most likely to differ or be uncorrelated on precisely the messy and difficult settings that are not captured in this stylized setting.
Critical weaknesses
-
I do not believe the analysis in section 6.2 is sufficient to suggest verbalized and internalized readouts access a shared representation. The authors appear to draw this conclusion from the fact that the residuals are correlated, presumably with the following reasoning, that if they did not access a shared representation then the readouts would only be correlated as much as they are with the true p_e. A claim involving representations would require experiments involving representations. There can be many reasons the residuals are correlated, any number of other shared factors would drive correlation without meaning shared representation.
-
I think the community probably wouldn't find the synthetic experiment surprising, nor the fine-tuning experiment (which is still quite synthetic). As the authors note in their discussion section, they do not capture "messy natural datasets" which I think is precisely the interesting setting to study, not to mention deception, sandbagging, etc which are not discussed at all.
Questions for authors
- can you provide more evidence to support your claim in 6.2?
- how far do you expect your results to generalize? i.e. what settings would you expect the readouts to align vs differ?
2 comments
Thank you for your clarifications and for the additional experiments in support of revised section 6.2.
I still do not find the new experiments sufficiently convincing for your revised claim of a "shared causal pathway". First, in the seed based partial correlation analysis, the partial correlation across seeds is already fairly high, and the increase within same seeds is modest in comparison. Does it make sense to also check the partial correlations when adding the confounders to the same seed version? Are the numbers in the table medians over (model, dataset) pairs and if so can you report the individual numbers with confidence intervals? Did you also check linearity assumptions to run the partial correlation?
For activation patching, if you are patching the entire entity representation, isn't it still possible that internalized vs verbalized readouts read from disjoint subspaces of that representation and use disjoint causal pathways? 29% vs 55% for patching fine-tuned into base also seems like a moderately large gap, how do you interpret this result?
For steering, can you provide more details on the training methodology and whether you handle potential overfitting given the high dimensional space? "For both probes, when the model is trained on frequency-based uncertainty, both readouts shift in proportion to the push and more than matched random and shuffled-target directions" Can you report the raw data?
Thank you for your thorough review. Your first weakness was brought up by several reviewers so we address it in a separate response here, where we rule out various confounders and also look at activation steering- and patching-based analyses. We respond to the rest of your comments and questions below. We are working on incorporating the changes into the manuscript, which we will update before the end of the rebuttal period.
[W2] Synthetic data/relevance to real datasets
Prior work has offered separate explanations for the two readouts: internal probabilities track relative frequencies in the data, and verbalized probabilities mimic human expressions of confidence and therefore predicts that the two readouts should be independent. We show that they are not: training through frequency-based or assertion-based data moves both readouts, mixing the two sources interpolates both, and, as the new interventions show, patching or steering the model's own activations moves both readouts.
We do show that readouts align on “real-world” data (Fig 2). However, since we do not have access to the data used to train most models, we are unable to cleanly explore the data-driven causes of this alignment. Our goal was to induce beliefs that are minimally confounded by the prior training data, which requires training data about facts that are not in the training data. Synthetic data is a clean way of achieving this.
Q1 can you provide more evidence to support your claim in 6.2?
See the general response here
Q2 how far do you expect your results to generalize? i.e. what settings would you expect the readouts to align vs differ?
We see robust coupling across both uncertainty sources and all three installation methods (LoRA, FFT, ICL), across 21 open-weights LLMs spanning four families, and under variation in level of detail. The readouts come apart in one case: for concise asserted statements, the verbalized readout almost perfectly tracks the installed belief while the internal readout is less aligned. Separately, for the smallest models and for detailed asserted statements under LoRA, both readouts align less well with the target. We focused on the axes of variation that we judged most likely to affect alignment, and we would rather not speculate beyond them; we will state the scope of our claims explicitly in the paper. We hope to see future work exploring the relationships between text data, training mechanisms, and model beliefs further, for example for beliefs acquired during pretraining or from naturally occurring data, and in larger models.