acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Intelligence from the Outside

Abstract

Inferring mechanisms underlying intelligence from observable behavior is a foundational challenge in artificial intelligence. We develop a theory of Bayesian intelligence for agents such as language models, which tests whether observed behavior could be generated by a rational agent responding to information revealed by inference-time reasoning. Each prompt induces a possibly imperfect internal experiment; the agent updates a full-support prior by Bayes' rule and faithfully reports its posterior over the possible answers to the question. Repetitions draw fresh, independent outcomes from the same unobserved experiment at one fixed state. We show that the agent's behavior admits this explanation if and only if its reports are not fully contradictory, i.e., some state remains possible under every report across all prompts. Report frequencies and the sizes of positive probabilities impose no further restrictions. We further propose and characterize the behavioral implications of one agent having access to a more informative experiment: there should exist a coupling of report distributions such that the more informative agent's report excludes every answer excluded by its counterpart. Finally, we show the difficulty of aggregating coarse reports: unless the agent reports a belief about the complete state of the world, the optimal aggregation can assign arbitrary weights to states that have not been excluded. These results provide a basis for understanding when agents' behavior can be modeled as revealing information through reasoning and highlight the difficulty of rejecting Bayesian rationality.

Primary area:
learning theory
Keywords:
Bayesian rationality, Blackwell experiments, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SvdAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper treats a trained LLM doing inference as doing Bayesian inference/updating. The authors model this with a full-support prior, i.e. the truth is always possible. The question then is about what happens when that trained LLM is prompted. The prompt induces an unobserved experiment, which causes the model to update and report a posterior over possible answers exactly. Each repetition of a prompt is an independent draw from that experiment at a fixed true state, so the user/observer only sees the distribution of reported vectors (of probabilities).

There are three main theoretical results/core ideas.

  1. Reports across prompts are Bayes-consistent iff some answer gets positive probability in every report across all prompts.
  2. An agent, say Agent 2, can plausibly have more informative reasoning than another agent, say Agent 1, iff Agent 2’s reports can be matched to Agent 1’s so that every answer Agent 1 rules out, Agent 2 also rules out. The authors call this plausible Blackwell dominance.
  3. Running the same prompt repeatedly and obtaining different probability vectors allows a characterization of how a Bayesian observer could combine those vectors without knowing the experiment. The idea here is that the combined belief must rule out any answer that any of the individual reports ruled out. This excludes simple averaging but allows for some pooling.

This is a theoretical paper. There is a worked example but no experiments and only a small amount of practical guidance.

Critical strengths

The main result shows that, from reported probability distributions alone, you can neither confirm nor reject that an LLM is doing Bayesian updating. I think this is a useful point for the empirical literature, which has made claims in both directions, and it makes clear what extra assumptions those tests are actually relying on.

Remark 6 is potentially practical for self-consistency style methods. Repeated samples are only independent given the full state (not the answer), so averaging probability vectors is not Bayes-consistent whenever supports differ.

Critical weaknesses

The model is only falsified by contradictory exact zeros across reports. It’s not clear how an LLM output maps to a report, and LLMs rarely output exact zeros. It seems that the main theorem is essentially vacuous in practice. This is perhaps the point of the paper though.

The cross-prompt condition only seems to matter if different prompts are about the same question Y_p and true state. It’s not clear who specifies the exact implied question Y_p or how to check that prompts share it.

Questions for authors
  1. How should LLM outputs be mapped to reports? When does an output count as zero?
  2. Who specifies Y_p in practice?
  3. Remark 1 requires the prior to give the truth positive probability. Is this reasonable for LLMs?
3 comments

Thank you for your careful reading and positive assessment. We agree with your reading of the paper. Indeed, one of the points we would like to convey is how hard it may be to test Bayesian intelligence in practice, as Theorem 1 points to logical contradictions as essentially the only criterion. (However, it still has empirical content, and we have observed LLMs outputting extreme values in practice.)

Regarding your questions:

  1. The simplest way to bring the model into practice is to send a prompt to an LLM that asks it to adhere to a certain JSON format in its output, culminating in an exact probability vector. We would read that vector as the belief. (If it is not, the “obedience” assumption of our model is violated.) Repeating exactly the same prompt in a brand new session would then be the simplest way to emulate the repetitions analyzed in the paper. Asking different prompts about several related variables would be another option. How to count zero beliefs is then an experimental choice. The cleanest requirement would be to allow for arbitrary reports and count only exact zeros. (If this is too permissive because of some output noise, then the prompt may request the answer to belong to a specified finite grid; such aggregation may limit the impact of noise.)

An example of such a prompt could be: “Estimate the probability that this patient has chronic kidney disease (CKD). Patient profile: {patient_profile} Briefly explain the main characteristics of the patient profile that inform your assessment. Then report your final probability that the patient has CKD. Output valid JSON on the final line with exactly this structure: {"pr_ckd": <number>} The number must be between 0 and 1. Your probability report will be evaluated by log score, so report your best probability estimate.”

  1. The easiest way to think about Y_p is as the variable being asked about directly in the prompt, in plain language. For example, in the prompt above, Y_p would be binary, indicating whether the patient has CKD.
  2. We view the “grain of truth” condition as being fairly minor. For example, it is automatically satisfied whenever a model has beliefs with full support, i.e., if it a priori allows for all possibilities. More generally, we expect modern LLMs to be trained on reasonable data and thus hold reasonable prior beliefs. However, let us highlight that Assumption 1 (correct specification) only strengthens our conclusions of fairly weak falsifiability of Bayesian intelligence. If the model is incorrectly specified, one would expect even weaker bounds on observable behavior.

We thank you again for your careful reading, and we hope our answers clarify these points.

Reviewer_SvdAOpenReview ↗

Response to author's comments

Thank you for the useful answers. My main concern is still the interpretation of exact zeros. Requesting structured probability reports addresses the output format, but not whether they actually reflect internal beliefs which is very unclear. Discretisation could also introduce artificial zeros rather than reveal real exclusions.

I also wonder whether a system like JEV, which produces probability estimates natively without necessarily performing Bayesian inference, could fulfil the same criteria. This seems to show the difference between showing that reports are compatible with Bayesian updating and establishing that the model actually reasons in this way.

Thank you. Indeed, our interpretation of exact zeros relies on the maintained Obedience assumption. If reports can differ arbitrarily from beliefs, then, in general, one cannot test for Bayesian updating of the unobserved beliefs from reports alone. Our framework only uses the observable outputs from a model, and no such framework could (even in principle) differentiate between a model that "truly" thinks in Bayesian terms and one whose outputs are consistent. Follow-up work could replace the Obedience assumption with a structural model of the discrepancy between beliefs and reports, or seek more direct access to the underlying beliefs, perhaps via mechanistic interpretability.

Regarding Jev in particular: Jev is potentially a less interesting test case for our theory because it does not perform any inference-time reasoning, and part of our motivation is to capture inference-time reasoning as a signal structure that reveals information to the model. One caveat is that the architecture for Jev is not public so we can't be sure, but open decision models are typically not stochastic. Regardless, an important distinction is that our framework is not concerned whether the model internally within each generation goes through a process that mechanistically corresponds to Bayesian reasoning, but rather whether the observable probability reports are Bayes-consistent (which they may very well be even if it's not obvious to us how the implementation works in mechanistic terms).

Reviewer_2aziOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors consider the problem of assessing the Bayesian rationalizability of a learner's revealed belief updates, as well as comparing (rationalized) information access of different learners by observing their revealed belief updates, and assessing the rationalizability of aggregations of independent experiment results. In all three cases, the authors present equivalent conditions for rationalizability.

Critical strengths

The paper is well-scoped and focused on a problem with rather fundamental value, i.e., the identifiability of a learner's information access without information access. The topic has become relevant given the super-human performance of AI systems in increasingly difficult domains, making human evaluators oftentimes operating without information access when facing AI systems with superior information or reasoning capacities. The authors also clearly execute the theoretical research with the target of obtaining insights, rather than producing fancy-looking results but induce little transferable learning in the reader, the latter being is increasingly worthless in light of AI's comparative advantage in producing them.

Critical weaknesses

For a theory aiming to inform the research and development of language models (defined by large-scale text-based, autoregressive training, with supervised + reinforced components, and meta-learning abilities) or agents (defined as AI systems taking actions in a stateful world), it is too detached from those systems. The theory is stateless, actionless, single-step, and without cross-episode learning, somehow dodging all the defining features of the systems that the abstract said the theory was developed for. Also, rationalizability (whether there exists some possible world where the agent is Bayesian) seems too weak a condition to be practically useful; to develop theoretical claims that can actually, e.g., "clarify when agreement across sampled answers predicts correctness and how those answers should be combined", one likely needs to leave the realm of sweeping no-free-lunch-style theorems (like Theorem 1-3 in this paper), and enter the realm of concrete models of bounded rationality of AI systems and humans, with both theoretical derivation and empirical validation.

If, instead, the theory is meant to be a general treatise on Bayesian rationalizability, it seems to be one addition to an existing pool of works discussing the same topic in different setups, i.e., the papers that the authors surveyed in "Bayesian Learning and Its Empirical Content". Theorem 1 is very standard in such a literature and is uninteresting; Theorem 2 is more interesting/insightful as it combines rationalizability and Blackwell dominance (I don't know of existing papers doing the same, but I may be missing things); Theorem 3 is less interesting as the definition of Bayes-plausible aggregate is itself very weak (h_T having positive probability), so it's unsurprising to see it get reduced to an equivalent weak condition. In general, the paper does bring some novelty to the topic of Bayesian rationalizability, but that novelty seems borderline in its extent.

Questions for authors

Do you see this paper as an application-motivated paper targeting the agent or LM context, or general theorizing about Bayesianism?

2 comments

Thank you for your careful reading and thoughtful review. We share your attitude towards producing fundamental insights and appreciate your positive assessment in this respect.

Regarding your question, our work is motivated by AI evaluation, while also contributing to the theory of Bayesian rationalizability. Specifically, many empirical studies assess the “rationality” of models under additional assumptions about what responses represent and how information is related across prompts or reasoning stages (e.g., Zhu & Griffiths 2024, Falck et al. 2024, He et al. 2025, Chen et al. 2026, and others discussed in Section 1.1). Our work clarifies what can be inferred when the internal experiments induced by prompts are unobserved and unrestricted, highlighting substantial challenges for empirical identification.

In addition, let us highlight that our current results are able to accommodate a broader range of settings than may initially appear. First, as noted in Remark 3, the results straightforwardly generalize to adaptive prompting. Second, they can also be easily generalized to settings with actions where the AI agents have known goals, i.e., known utility functions. For example, the experimenter could supply a utility function in the prompt and the “obedience” assumption would entail that the agent takes an action which maximizes the specified utility given its posterior belief. In this case, each action taken by the agent can be interpreted as a report of (a range of) beliefs at which this action is optimal, and then all of the existing analysis can be applied to those reported beliefs.

At the same time, considering the challenges raised by AI leads to novel contributions to the theory of Bayesian intelligence. As we argue in the paper, it is essential to systematically analyze behavior conditional on a fixed state and our contribution is to characterize the joint feasibility of coarse reports across prompt-dependent experiments. As such, while we agree with your appreciation of Theorem 2, we view Theorem 1 as a useful contribution to the existing literature. That theorem shows that “logical” consistency of responses is necessary and sufficient for Bayesian rationalizability conditional on a fixed state, without further restrictions on report frequencies or the sizes of positive probabilities, which are often required. Similarly, we also find Theorem 3 informative. Although the aggregation restriction is weak, this weakness is itself informative: as we discuss in Remark 6, full-state reports and a known prior uniquely determine the Bayesian aggregate under conditional independence, whereas coarse reports generally do not. As such, it highlights the critical role of the informational bottleneck in belief reports, as well as the importance of carefully defining the state space in such an analysis.

Thank you for the reply. I think we agree on the implications of the theory, but I still hold my reservations on the significance/importance of these implications.

I will keep my score for now, but I'd like to highlight to the AC that this is very much a borderline case for me.

Thank you.

Reviewer_JZ8cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the limits of representing an LLM as a Bayesian agent. It essentially asks the rationalizability question: if we observe answers produced by an LLM, what report sequences of answers are consistent with this LLM acting as a perfect Bayesian agents when the unknown state of the world is fixed. The paper’s theorem 1 answers this question. The paper then goes further and ask: if we have two LLMs, when do we know from their observed answers that one LLM is more informative than another LLM in the Blackwell ordering sense (this is answered by Theorem 2). Furthermore, the authors ask if we don’t know how the LLM works (i.e. don’t know the interal Bayesian experiment of the LLM), can repeated answers from the LLM tells us how to optimally aggregate these answers (answered in Theorem 3).

Critical strengths

The paper has a very simple and clean conceptual model, and the set of results is very intuitive. I view this paper as an alert to claims that LLMs behave as Bayesian agents. Because if they indeed are, then this paper shows that it’s a very weak statement without knowing how the internal reasoning works. It shows that Bayesian rationalizability provides little evidence about the internal reasoning. I think this is a meaningful first step. The paper is well written and easy to follow.

Critical weaknesses
  • Once the model is set up, the results follow straightforwardly.

  • Perfect Bayesian rationality is a very strong requirement that no one expects either humans or LLMs to satisfy. The set of results applies in the same way to LLMs and to humans. So, in some sense, the results are not surprising. The impossibilities are more impossibilities to perfect Bayesian reasoning as a model for agents.

Questions for authors

I wonder whether the authors have thought about how to allow deviations from a perfect Bayesian model. It won’t give such sharp results, but it’s a more realistic model.

2 comments

Thank you for your careful reading and support, we really appreciate it. We agree with your overall assessment and view the paper as a step towards a rigorous and comprehensive analysis of intelligence in AI systems. We would just like to add brief clarifications regarding the “weakness” points. First, we agree that the results, once the model is set up, do not require mathematical apparatus that would go beyond that used in existing analyses in the related literature (although we wouldn’t say that the proofs are trivial either). As such, we view the main value as lying in creating the right model and posing the right questions. Second, we agree that perfect Bayesian rationality is a strong requirement. However, the strength of this requirement makes the fairly weak empirical conclusions only more surprising, not less. Perfect rationality is also a natural aim for intelligent systems, and, as you note, studying it is a necessary first step.

Regarding your question, we believe there could be many productive ways to extend the model toward bounded rationality. But while we considered this, let us note that one of the perhaps surprising takeaways of this model is that Bayesian rationality is fairly hard to reject, and we expect that allowing for bounded rationality would only strengthen that conclusion.

Thank you for your response!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.