- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper treats a trained LLM doing inference as doing Bayesian inference/updating. The authors model this with a full-support prior, i.e. the truth is always possible. The question then is about what happens when that trained LLM is prompted. The prompt induces an unobserved experiment, which causes the model to update and report a posterior over possible answers exactly. Each repetition of a prompt is an independent draw from that experiment at a fixed true state, so the user/observer only sees the distribution of reported vectors (of probabilities).
There are three main theoretical results/core ideas.
- Reports across prompts are Bayes-consistent iff some answer gets positive probability in every report across all prompts.
- An agent, say Agent 2, can plausibly have more informative reasoning than another agent, say Agent 1, iff Agent 2’s reports can be matched to Agent 1’s so that every answer Agent 1 rules out, Agent 2 also rules out. The authors call this plausible Blackwell dominance.
- Running the same prompt repeatedly and obtaining different probability vectors allows a characterization of how a Bayesian observer could combine those vectors without knowing the experiment. The idea here is that the combined belief must rule out any answer that any of the individual reports ruled out. This excludes simple averaging but allows for some pooling.
This is a theoretical paper. There is a worked example but no experiments and only a small amount of practical guidance.
Critical strengths
The main result shows that, from reported probability distributions alone, you can neither confirm nor reject that an LLM is doing Bayesian updating. I think this is a useful point for the empirical literature, which has made claims in both directions, and it makes clear what extra assumptions those tests are actually relying on.
Remark 6 is potentially practical for self-consistency style methods. Repeated samples are only independent given the full state (not the answer), so averaging probability vectors is not Bayes-consistent whenever supports differ.
Critical weaknesses
The model is only falsified by contradictory exact zeros across reports. It’s not clear how an LLM output maps to a report, and LLMs rarely output exact zeros. It seems that the main theorem is essentially vacuous in practice. This is perhaps the point of the paper though.
The cross-prompt condition only seems to matter if different prompts are about the same question Y_p and true state. It’s not clear who specifies the exact implied question Y_p or how to check that prompts share it.
Questions for authors
- How should LLM outputs be mapped to reports? When does an output count as zero?
- Who specifies Y_p in practice?
- Remark 1 requires the prior to give the truth positive probability. Is this reasonable for LLMs?
3 comments
Response to author's comments
Thank you for the useful answers. My main concern is still the interpretation of exact zeros. Requesting structured probability reports addresses the output format, but not whether they actually reflect internal beliefs which is very unclear. Discretisation could also introduce artificial zeros rather than reveal real exclusions.
I also wonder whether a system like JEV, which produces probability estimates natively without necessarily performing Bayesian inference, could fulfil the same criteria. This seems to show the difference between showing that reports are compatible with Bayesian updating and establishing that the model actually reasons in this way.
Thank you. Indeed, our interpretation of exact zeros relies on the maintained Obedience assumption. If reports can differ arbitrarily from beliefs, then, in general, one cannot test for Bayesian updating of the unobserved beliefs from reports alone. Our framework only uses the observable outputs from a model, and no such framework could (even in principle) differentiate between a model that "truly" thinks in Bayesian terms and one whose outputs are consistent. Follow-up work could replace the Obedience assumption with a structural model of the discrepancy between beliefs and reports, or seek more direct access to the underlying beliefs, perhaps via mechanistic interpretability.
Regarding Jev in particular: Jev is potentially a less interesting test case for our theory because it does not perform any inference-time reasoning, and part of our motivation is to capture inference-time reasoning as a signal structure that reveals information to the model. One caveat is that the architecture for Jev is not public so we can't be sure, but open decision models are typically not stochastic. Regardless, an important distinction is that our framework is not concerned whether the model internally within each generation goes through a process that mechanistically corresponds to Bayesian reasoning, but rather whether the observable probability reports are Bayes-consistent (which they may very well be even if it's not obvious to us how the implementation works in mechanistic terms).
Thank you for your careful reading and positive assessment. We agree with your reading of the paper. Indeed, one of the points we would like to convey is how hard it may be to test Bayesian intelligence in practice, as Theorem 1 points to logical contradictions as essentially the only criterion. (However, it still has empirical content, and we have observed LLMs outputting extreme values in practice.)
Regarding your questions:
An example of such a prompt could be: “Estimate the probability that this patient has chronic kidney disease (CKD). Patient profile: {patient_profile} Briefly explain the main characteristics of the patient profile that inform your assessment. Then report your final probability that the patient has CKD. Output valid JSON on the final line with exactly this structure: {"pr_ckd": <number>} The number must be between 0 and 1. Your probability report will be evaluated by log score, so report your best probability estimate.”
We thank you again for your careful reading, and we hope our answers clarify these points.