acceptodds
Under review as a conference paper at ICLR 2027

BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation

Abstract

Large language models (LLMs) are typically deployed in multi-turn conversations, where each turn provides new evidence that should reduce epistemic uncertainty about their environment. Acting rationally then requires inferring the unobserved quantities that govern it and updating beliefs about them as evidence accumulates. Yet most evaluations only score the model's final-turn answer in a single-turn format, leaving this process unexamined. We ask how closely LLMs' belief updates match those of a rational Bayesian reasoner in multi-turn settings, and introduce BayesBench, a suite of simulation environments that probe this across three progressively complex tasks: (i) Bayesian estimation, where the model infers an unknown parameter from sequential evidence; (ii) Bayesian prediction, where the model turns inferred beliefs about a latent variable into outcome forecasts; and (iii) latent-framed Bayesian prediction, where observations are filtered through a user-persona framing, requiring joint inference over the latent state and the persona. Across seven LLMs (3B–70B), scaling improves latent inference and evidence accumulation, with updates occasionally matching the Bayesian posterior. However, these gains do not reliably carry over to downstream prediction, exposing a gap between inferring latent structure and using it to rationally update beliefs about the target outcome.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Multi-turn Conversations, Belief Updating, Bayesian Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zwSFOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how LLMs update their beliefs during multi-turn interactions where evidence arrives sequentially. It introduces BayesBench, a evaluation suite structured around three core Bayesian tasks instantiated in four distinct environments: (1) Bayesian estimation via a coin-flip task with unknown bias; (2) Bayesian prediction via a cold-start recommender system predicting held-out movie ratings from sequential ratings; and (3) latent-framed Bayesian prediction via social judgment (r/AmItheAsshole) and medical triage tasks, where observations are filtered through a user's communication style (latent persona).

Critical strengths

Rigorously Controlled Multi-Turn Evaluation Framework: BayesBench moves beyond standard single-turn final-answer evaluations by tracking intermediate belief trajectories across sequential turns. The integration of closed-form Bayesian references (for estimation and recommendation tasks) alongside debiased multi-choice option probing via Latin squares provides a methodologically sound and mathematically grounded way to isolate belief-updating dynamics from simple output formatting artifacts.

Identification of the "Inference-Prediction Gap": The paper uncovers a valuable and counter-intuitive insight: while model scaling reliably improves latent variable recovery (e.g., identifying user types or patient communication styles), this capability does not automatically yield better-calibrated downstream predictions. This exposes a critical limitation in current LLM architectures and post-training methods regarding how implicit latent information is operationally utilized.

Critical weaknesses

Reliance on Constrained and Discrete Latent Structures: The simulation environments heavily rely on simplified, highly discretized categorical spaces (e.g., K=4K=4 movie-viewer profiles or fixed communication styles). Real-world user interactions feature continuous, mixed, and dynamically shifting preferences that overlap in complex ways, potentially limiting how well these findings generalize to organic human-chatbot dialogues.

Confounding Effects of LLM-Simulated Interlocutors: In the active engagement tasks (social judgment and medical triage), user dynamics are generated by LLM simulators following fixed style prompts. This introduces potential artifacts or biases inherent to the simulator models themselves (such as predictable framing styles or stylized lexical choices), which may not fully capture the chaotic and irregular feedback loops of true human interlocutors.

Inherent Limitations of Multiple-Choice Probing: Belief elicitation relies on extracting token log-probabilities over closed-form multiple-choice options (via Latin square counterbalancing). While this controls for position bias, it forces a discrete format that may restrict the model's expressive capacity and fail to completely mirror internal confidence or beliefs that would otherwise emerge in free-form, unconstrained text generations.

Questions for authors

Clarification Request: In the latent-framed tasks (social judgment and medical triage), explicit conditioning on the model's argmax inferred latent improved downstream prediction accuracy for certain model scales (e.g., Figures 4 and 7). Given that explicit prompting bridges part of the gap, have you experimented with reinforcement learning or fine-tuning techniques (such as training on intermediate Bayesian reasoning traces) to close the gap natively without requiring manual latent injection at inference time?

Suggestion for Future Work: It would be valuable to understand how proprietary frontier models (e.g., GPT-4 or Claude 3.5 Sonnet) perform on BayesBench compared to the open-weight models evaluated here, particularly regarding whether scale alone in massive closed models mitigates extreme over-updating or if it represents a fundamental limitation of next-token prediction architectures.

2 comments

We thank the reviewer for describing the evaluation as rigorously controlled and the inference–prediction gap as a valuable and counter-intuitive finding. We have added a Bayesian reference given exactly the LLM's information (response to Reviewer fhBu, Q1), and credible intervals for all main claims. Against this reference, larger models still infer the user's type better while their rating predictions move further away, and the credible intervals for both trends exclude zero. We address each point below.

W1. "The simulation environments heavily rely on simplified, highly discretized categorical spaces … potentially limiting how well these findings generalize to organic human-chatbot dialogues."

A1. We agree that organic dialogues are richer than our environments, and our limitations discuss this (App. B). The structure is deliberate: scoring the model's belief at every turn needs a known likelihood, so that a Bayesian reference, or conditions any Bayesian reasoner must satisfy, can be computed. Within this constraint, the design is not limited to categories: The generative process in Sec. 2.1 places no restriction on the latent, which is continuous in Task 1 (the coin's bias) and categorical in Tasks 2–3, and the same template allows continuous or mixed latents, which we list as future work. The environments also move toward organic dialogue step by step. The recommender's user types are fitted to real MovieLens ratings, and social judgment is built on real r/AmItheAsshole posts; in social judgment and medical triage, each case is delivered over many turns in different communication styles, and we test conditions any Bayesian reasoner must satisfy instead of an exact posterior (response to Reviewer fhBu, W3). The inference–prediction gap appears even in the controlled recommender, where the reference is exact.

W2. "…user dynamics are generated by LLM simulators following fixed style prompts. This introduces potential artifacts or biases inherent to the simulator models themselves…"

A2. We agree, and our limitations discuss this (App. B). In terms of Eq. (1), the simulator implements only the framing of each observation, ℓηf(xt∣x~t,zf)\ell_{\eta_f}(x_t \mid \tilde{x}_t, z_f), which limits what it can introduce:

  • The facts are fixed. Each case is decomposed into ordered clean segments x~t\tilde{x}_t, and the simulator only rewords the current one in the assigned communication style; its private steering instructions are stripped before the evaluated model sees the context. Under passive observation, the model sees the clean segments directly.
  • Style comparisons hold the simulator fixed, so artifacts common to all styles cancel.
  • Simulator failures are documented. App. G.4 lists the persona failure modes we found and how the steering protocol handles them.

W3. "…it forces a discrete format that may restrict the model's expressive capacity and fail to completely mirror internal confidence or beliefs that would otherwise emerge in free-form, unconstrained text generations."

A3. Fixed-choice probing gives a full distribution over a fixed set of outcomes, which we need in order to compare with the Bayesian reference at every turn. It also avoids splitting probability across different phrasings of the same answer, a limitation Gupta et al. note; fully free-form answers would have to be mapped back to the outcomes, which reintroduces this problem. To test whether the result depends on the probe, we had the models state a percentage for each option in words, on the same conversations (response to Reviewer cacN, W2). The two readouts differ, but under both, type beliefs improve with scale and rating predictions do not move toward the reference; the stated ratings are broad but barely respond to the evidence. We keep token probabilities as the primary readout because they are the distribution the model's answers are generated from; Pal et al. (2026) find that elicited confidences often fail to predict actions.

Q1. "…have you experimented with reinforcement learning or fine-tuning techniques … to close the gap natively without requiring manual latent injection at inference time?"

A4. We have not; training is out of scope for this work, but it is a very interesting direction. Qiu et al. show that fine-tuning on a Bayesian assistant improves interactive recommendations, so training toward Bayesian behavior is feasible; our results suggest targeting the step from inferred latent to prediction. As you note, latent-conditioned prediction shows that some models can use the inferred latent when it is written into the prompt, though the gains are modest and uneven; training would aim for the same without that injection. BayesBench measures this directly: training succeeds to the extent that the prediction without the injected latent approaches the prediction with it. To keep the benchmark an evaluation, training would use separately generated environments rather than the benchmark's own.

Q2. "It would be valuable to understand how proprietary frontier models … perform on BayesBench compared to the open-weight models evaluated here…"

A5. Our probe needs token log-probabilities for the answer options, which current frontier APIs return only for some models and settings (for example, not for reasoning models), as Gupta et al. also note. Asking models to state probabilities in words would avoid this. In our test of that (response to Reviewer cacN, W2), stated beliefs about the user's type responded to the evidence, but stated rating predictions stayed nearly the same whatever the evidence. Stated probabilities therefore cannot yet replace token probabilities for rating prediction, and evaluating closed models needs a better way to elicit those predictions, which we list as future work.

We hope these additions, including the test of the fixed-choice probe against stated probabilities (A3), address your concerns, and would be grateful if you would please consider raising your score. Thanks so much!

Gupta et al. (ACL 2025), "Enough Coin Flips Can Make LLMs Act Bayesian." Qiu et al. (Nature Communications, 2026), "Bayesian teaching enables probabilistic reasoning in large language models." Pal et al. (2026), "Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions."

Reviewer_cacNOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors build a benchmark for evaluating the ability of language models to perform Bayesian inference in context. Tasks in the benchmark are categorized by the topology of the causal diagram connecting hidden variables and observations. The authors evaluated models along a ladder of increasing indirectness of observation, and discovered factors impacting models' ability to perform Bayesian inference.

Critical strengths

The conceptual framework of evaluating model's Bayesian inference capability (and by extension, the capability of applying Bayesian rationality for expected utility maximization) through a ladder of increasingly indirect causal topologies is interesting and potentially generalizable. I could imagine such a framework be extended to quantify model's broader capability of acting under uncertainty, where the (increasingly sophisticated) causal diagrams provide experimenters with means of data generation. Also, quantifying the level of indirectness in any given diagram, and quantitatively study how model's inference capability changes with increasing indirectness, would also inform real-world inference where indirectness is high.

The choice of tasks is great; they cleanly fit the causal diagrams, and the authors have done a good job avoiding potential confounds.

Critical weaknesses

The empirical findings need more sensitivity/robustness tests; there are a range of factors that may significantly change the findings.

  • Way of conveying the conditional likelihood function to the model. For the biased coin experiment no such conveying is needed except for saying it's a biased coin (the model knows what a biased coin is). For the recsys case, for example, things become tricker, e.g. would there be any difference in results if we convey the conditional likelihood function not by laying out the function's value table, but through randomly sampled many-shot examples? Models often exhibit different behaviors when told something verbally vs. when learning the same thing through in-context examples.
  • Verbal reports vs token probs. Models, especially instruction-tuned models, often behave extremely differently when being asked to verbally report probabilities vs when their output token probs are directly observed. In particular, the latter tends to contain extremely large biases introduced by instruction training. Are similar effects observed here?
  • (Optional) Base model vs instruction-tuned mode. One might expect pretrained base models perform Bayesian inference better than instruction-tuned models.

Also: statistical tests are missing from most of the findings.

Questions for authors

Please see "critical weaknesses".

4 comments

We thank the reviewer for the constructive review and for seeing the ladder of increasingly indirect causal structures as a generalizable way to study inference under uncertainty. We address each point below. Brackets are 95% credible intervals (App. C.2).

W1. "…would there be any difference in results if we convey the conditional likelihood function not by laying out the function's value table, but through randomly sampled many-shot examples?"

A1. The prediction result is unchanged; the inference result changes. The paper's prompt gives each user type's share of users and its average rating per item as a table. To test the format, we replaced the table with five sampled example users per type, each rating all 50 items, with everything else unchanged. We ran Qwen 2.5 at 3B, 14B and 32B (32 trials each, on the paper's rating sequences) and compared each against a Bayesian reference given the same examples.

TVD ×100 to the reference given the same examplesQwen-3BQwen-14BQwen-32BPer doubling
Type posterior57.354.261.0+0.8 [−0.2, +1.8]
Rating predictive35.445.859.8+6.8 [+6.3, +7.3]
Prob. on top rating (%)526480

As with the table, larger models' rating predictions move further from the reference and become more concentrated. The inference result depends on the format. With the table, larger models identify the user's type more accurately. When the types must instead be learned from five example users each, they do not: at later turns, Qwen-32B picks the true type 28% of the time (chance: 25%).

In terms of the generative process in Sec. 2.1, the examples require the model to estimate the likelihood ℓη\ell_\eta from samples before it can infer HH, rather than being given each type's average ratings. This is a meaningfully harder task: even the Bayesian reference identifies the true type 60% of the time from the examples, against 79% from the table. Larger models do not handle this added step better here. We have added this condition to App. F.1.2.

W2. "Models, especially instruction-tuned models, often behave extremely differently when being asked to verbally report probabilities vs when their output token probs are directly observed … Are similar effects observed here?"

A2. The readouts differ substantially, as you anticipated, but the main result holds under both. We replayed the same recommender conversations and asked the same questions a second way: the model states a percentage for each option (greedy decoding), averaged over the same option rotations. Both readouts use identical contexts (Qwen 2.5 at 3B, 14B and 32B; 32 trials each) and are scored against a Bayesian reference built from exactly the information in the prompt (described in our response to Reviewer fhBu, Q1).

TVD ×100 to the matched referenceQwen-3BQwen-14BQwen-32BPer doubling
Type posterior, token54.245.141.8−3.7 [−4.6, −2.8]
Type posterior, stated44.342.034.2−2.7 [−3.1, −2.3]
Rating predictive, token28.857.566.8+11.3 [+11.0, +11.7]
Rating predictive, stated17.719.422.2+1.3 [+1.1, +1.5]

Under both readouts, type beliefs improve with scale, and rating predictions do not get closer to the reference. The stated ratings score a lower TVD, but they are a broad distribution that barely changes with the evidence: repeating each model's average stated answer at every turn scores about the same (16.0, 19.8 and 22.4), because the reference is itself broad. The token ratings track the evidence but become over-concentrated (41%, 68% and 82% on the top rating, against 31%). Neither matches the Bayesian prediction, which is both broad and specific to the evidence: under either readout, the rating prediction does not reflect the improving type belief. We use token probabilities as the primary readout because they are the distribution the model's answers are generated from; Pal et al. (2026) find that LLMs' elicited confidences often fail to predict their actions. We have added this comparison to App. F.1.2.

W3. "(Optional) Base model vs instruction-tuned mode. One might expect pretrained base models perform Bayesian inference better than instruction-tuned models."

A3. We evaluate instruction-tuned models because the benchmark's main condition is a multi-turn chat. Gupta et al. find that base models (Pythia, up to 12B) broadly follow Bayesian updates in a coin-flip task, which is consistent with your intuition. Their models are, however, smaller than those where we see the strongest over-updating (Qwen-32B and LLaMA-70B), so this does not settle whether instruction tuning introduces it. Comparing instruction-tuned models with their base models would help pinpoint whether the departures from Bayesian updating we observe emerge in pretraining or are introduced by instruction tuning; the coin-flip task, which needs no chat format, is a natural place to start.

W4. "Also: statistical tests are missing from most of the findings."

A4. Thank you for noting this. App. C.2 now reports 95% credible intervals for every main claim. Scaling claims are tested as the change per doubling of model size within each family (e.g., the rating predictive TVD in A1 rises by 6.8 [6.3, 7.3] per doubling), and engagement claims as the paired difference between passive observation and active engagement on the same cases (e.g., for Qwen-3B, the probability of the correct urgency in triage drops by 0.17 [0.14, 0.20] under active engagement). Where an interval includes zero, as for the type posterior in A1, we do not claim an effect.

We hope these additions, the two robustness experiments on how the likelihood is conveyed and how beliefs are elicited, and statistical tests for all main claims, address your concerns, and would be grateful if you would please consider raising your score. Thank you so much!

Gupta et al. (ACL 2025), "Enough Coin Flips Can Make LLMs Act Bayesian." Pal et al. (2026), "Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions."

I think these additions are satisfactory and I am increasing my score. Please make sure to discuss them in your revision.

Thank you so much for taking the time to review our work and rebuttal! We're glad that the additions were well received and very much appreciate you increasing your score to reflect that. We will be sure to discuss them in our revision.

Reviewer_fhBuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BayesBench studies how LLMs revise their predictions as evidence arrives over multiple turns. It evaluates four settings: estimating a coin’s bias from successive flips, inferring a user’s preferences from movie ratings, judging an unfolding social conflict, and assessing medical urgency from symptom descriptions. Its central question is whether models can both infer relevant hidden information, e.g. a user’s preference type or a patient’s communication style, and use that information to make better predictions. Across seven models, the authors find that larger models often identify these hidden variables more successfully, but downstream predictions do not improve consistently. Overall, the paper offers a new evaluation on LLM's decision making from the Bayes perspective.

Critical strengths
  • The topic of such explicit distinction between inference and prediction is useful for understanding LLM's decision making. A model may correctly identify a user’s preferences yet still recommend unsuitable items. Evaluating these separately, and following their development throughout an interaction, can reveal failures that final-answer accuracy alone would not suffice.
  • The introduced benchmark is diverse on domains. The coin and recommendation environments enable comparison with explicit Bayesian references, while the social and medical tasks explore how communication style affects predictions. These orthogonal dimensions could offer insightful findings on different scenarios.
Critical weaknesses
  • My main concern is that as an evaluation paper, quite of the experimental settings and conceptual findings are not unique to this work, e.g. Gupta et al. investigate sequential coin-flip predictions against Bayesian references; Qiu et al. study preference learning through repeated user interactions and show that recommendations computed externally from elicited preferences can outperform the model’s direct recommendations; BASIL examines whether socially induced belief shifts are consistent with Bayesian updating. BayesBench differs in particular tasks and trajectory measurements, but it needs to be more precise on what this work establishes beyond those existing results.
  • The main findings in this work are interesting but does not surface a distinctive insight on how models fail. Imperfect Bayesian behavior, uneven improvements with model size, and sensitivity to conversational framing are all consistent with previous evidence. A broader benchmark can nevertheless be valuable if it reliably distinguishes different failure modes. Here, it remains unclear whether the four environments expose a common failure to use inferred information or several different problems, such as probability elicitation, missing task knowledge, and prompt sensitivity. In the social and medical settings, there is also no exact Bayesian posterior against which to assess the updates. The results therefore support a collection of behavioral observations more clearly than a unified pattern of Bayesian reasoning.
  • The uncertainty of LLM itself can be difficult to quantify. More clarification/analysis on separating model uncertainty can strengthen the significance of the conclusions.

Gupta et al. (ACL 2025), "Enough Coin Flips Can Make LLMs Act Bayesian." Qiu et al. (Nature Communications, 2026), "Bayesian teaching enables probabilistic reasoning in large language models." Atwell et al., "BASIL: Bayesian Assessment of Sycophancy in LLMs."

Questions for authors
  • Does the inference–prediction gap persist when the LLM and Bayesian reference receive identical information, including the target item’s relationship to each user type?
4 comments

We thank the reviewer for the careful review and for recognizing that separating latent inference from prediction, and tracking both over an interaction, can reveal failures that final-answer accuracy misses in conventional evals. We address each concern below. Brackets are 95% credible intervals (App. C.2).

W1. "…quite of the experimental settings and conceptual findings are not unique to this work … it needs to be more precise on what this work establishes beyond those existing results."

A1. LLMs are increasingly deployed in multi-turn interactions, where evidence arrives sequentially and is often shaped by hidden quantities that must be inferred from observations. Acting rationally then requires inferring these quantities and updating beliefs about them as evidence accumulates. BayesBench poses this as three progressively complex tasks: Bayesian estimation, where the model infers an unknown parameter from sequential evidence; Bayesian prediction, which adds a latent that the model must infer to guide a downstream judgment; and latent-framed Bayesian prediction, which further filters the evidence through a framing latent (the semantic style in which it is presented) that the model must infer and condition on. Each of the three works studies part of this structure:

  • Gupta et al. study only Bayesian estimation: the entire paper is a coin-flip experiment, where the latent (the coin's bias) is itself the target, so inferring latent structure and using it cannot come apart. In BayesBench, this is the first of three tasks.
  • Qiu et al. study Bayesian prediction, but their evaluation focuses on downstream agreement with the assistant's choices, not the beliefs behind them.
  • BASIL uses a Bayesian framework to distinguish sycophantic shifts from rational responses to evidence, for an opinion the user states in a single update, not a framing latent inferred over turns.

Our three findings each go beyond these works:

Models infer latent structure from accumulating evidence, and scaling strengthens this inference. Smaller models adapt too weakly to the observed distribution: LLaMA-3B's coin prediction stays at 0.49–0.55, and in triage the smallest models stay much closer to chance at identifying the patient's communication style. Larger models discern the latent but can over-update toward the extremes when the evidence is skewed: for two coins where the reference is at 0.24 and 0.72, Qwen-32B and LLaMA-70B predict near 0 and near 1. Gupta et al. find little effect of scale (Pythia base models up to 12B) and attribute residual error to the prior rather than the update. Six of seven models start near the reference's 50% heads, so in our setting the departure is in the update; our results suggest that this alignment is highly circumstantial in current models, even though the underlying capability is present.

Better latent inference does not yet translate into calibrated downstream prediction. As more ratings accumulate, larger models recover the user's latent type more accurately, yet their rating predictions move away from the Bayesian reference: LLaMA-70B and Qwen-32B put 69% and 84% on one rating, against the reference's 31% (A5). This gap cannot arise in Gupta et al.'s setting, where the latent is the target, and agreement with the assistant's choices cannot detect it. Qiu et al.'s appendix shows that recommendations derived from the model's stated preferences can beat its direct ones; we show that this gap widens with scale.

Active engagement can bias how models accumulate evidence. BASIL concludes that a user's stated opinion moves the model's posterior more than the same opinion from a third party, in a single update. We ask whether engagement alone shifts beliefs over many turns. With the user reporting the facts neutrally, active engagement shifts final beliefs relative to passive observation of the same evidence: toward the user in social judgment, and away from the correct urgency in triage for five of seven models. In social judgment, larger models reduce this bias as evidence accumulates.

We have expanded this comparison in App. A.

W2. "…it remains unclear whether the four environments expose a common failure to use inferred information or several different problems, such as probability elicitation, missing task knowledge, and prompt sensitivity."

A2. The four environments expose a common failure: models increasingly infer latent structure, but fall short of using it to rationally update beliefs about the target outcome. The formalization in Sec. 2.1 makes this precise.

All four environments instantiate the generative process in Sec. 2.1: H∼πηH \sim \pi_\eta, Y∣H=h∼gη(⋅∣h)Y \mid H = h \sim g_\eta(\cdot \mid h), and Xt∣H=h,X1:t−1=x1:t−1∼ℓη(⋅∣h,x1:t−1)X_t \mid H = h, X_{1:t-1} = x_{1:t-1} \sim \ell_\eta(\cdot \mid h, x_{1:t-1}), where HH is the hidden quantity the model must infer, YY is the prediction target (absent in Task 1), and η\eta is the fixed environment specification. Given the latent hh, the observations are conditionally independent of the target, so the posterior over HH and the predictive over YY are linked by marginalization:

pη(y∣x1:t)=∫pη(h∣x1:t) gη(y∣h) dh.p_\eta(y \mid x_{1:t}) = \int p_\eta(h \mid x_{1:t})\, g_\eta(y \mid h)\, dh.

In Task 1 (coin flip), H=ΘH = \Theta is the coin's bias. In Task 2 (recommender), H=ZH = Z is the user's type, which must be used through this marginalization to predict the held-out rating YY. In Task 3 (social judgment, medical triage), H=(Zs,Zf)H = (Z_s, Z_f) decomposes into a state ZsZ_s, the underlying scenario, and a framing ZfZ_f, a communication style. Since gη(y∣zs,zf)=gηs(y∣zs)g_\eta(y \mid z_s, z_f) = g_{\eta_s}(y \mid z_s), the same case should yield the same verdict regardless of framing, so the model must infer ZfZ_f and condition on it. We elicit the model's posterior over the latent and its predictive over YY separately (Sec. 2.2), which lets us place each environment's failure mode:

  • Coin flip. Smaller models adapt too weakly to the observed distribution, and larger models over-update toward the extremes (A1). With YY absent, this isolates the update itself: larger models turn evidence into beliefs more extreme than it supports.
  • Recommender. Larger models recover the user's latent type more accurately, yet their rating predictions move further from the marginalization (A5). Inference improves; its use does not.
  • Medical triage. The strongest models reach about 0.80 probability on the patient's true communication style mid-conversation, but fall short of reliably using the inferred reporting style to calibrate their urgency decisions: latent-conditioned prediction gives modest and uneven gains, the middle urgency tiers are pushed toward the extremes, and active engagement lowers the probability of the correct urgency (Qwen-3B: by 0.17 [0.14, 0.20]).
  • Social judgment. The framing should not move the verdict, yet active engagement shifts beliefs toward the user relative to passive observation of the same evidence (Qwen-3B: the probability that the user is at fault drops by 0.49 [0.37, 0.61]).

We thank the reviewer for raising alternative explanations. Each corresponds to a component of this generative process, which lets us check whether it accounts for these results:

  • Probability elicitation would be expected to affect the elicited posterior and predictive alike. Yet the same probe, averaged over rotations of the option-to-letter mapping to cancel position bias, reads type beliefs that move toward the reference with scale and rating predictions that move away (A5). When models instead state probabilities in words, type beliefs still improve with scale and rating predictions still do not (response to Reviewer cacN, W2).
  • Missing task knowledge would appear as a wrong likelihood ℓη\ell_\eta or gηg_\eta. Recommender items are anonymized, so world knowledge cannot help, and when we give the model each type's average rating for the target item, leaving only the marginalization, the gap persists (A5). In triage, each new symptom moves the urgency belief in the expected direction (App. H.1, Fig. 15): models interpret each piece of evidence correctly, and the failures arise when they combine it.
  • Prompt sensitivity corresponds to the framing ZfZ_f, which a Bayesian reasoner infers and conditions on. With the facts fixed, the user's communication style changes the final verdict error in social judgment by less than 0.05 on average. Where the framing does move beliefs, the shift has a consistent direction and, in social judgment, shrinks as larger models accumulate evidence (A1). This points to a failure to condition on the framing rather than to sensitivity to wording.

We therefore read the four environments as exposing one failure, in using inferred information to update beliefs about the target outcome, rather than several unrelated problems.

W3. "In the social and medical settings, there is also no exact Bayesian posterior against which to assess the updates."

A3. In these environments we instead test conditions that the generative process (Sec. 2.1) implies for any Bayesian reasoner (A2):

  • Identical facts should yield the same belief, whether observed passively or in conversation, since the conversation adds no information. This fails in both environments (A2).
  • The user's style should not shift the answer, since it carries no information about the correct answer. This holds in social judgment; in triage, models fall short of correcting for the inferred style (A2).
  • Each new symptom should move the urgency belief in the direction it indicates. This holds on average (App. H.1).

The over-extreme triage predictions (A2) also match what we measure against an exact posterior in the coin and recommender.

W4. "The uncertainty of LLM itself can be difficult to quantify. More clarification/analysis on separating model uncertainty can strengthen the significance of the conclusions."

A4. Our probe gives a full distribution at every turn, so we can compare its spread with the Bayesian reference's. In the recommender, we read the type belief and the rating prediction separately and compare each with the reference. Larger models' type beliefs approach the reference, while their rating predictions become far more concentrated than it (A5).

Q1. "Does the inference–prediction gap persist when the LLM and Bayesian reference receive identical information, including the target item's relationship to each user type?"

A5. It does. In our recommender, the prompt does not give each type's average rating for the target item. To keep the information identical, we rebuilt the Bayesian reference from exactly what the prompt gives the LLM: each type's share of users, and each type's average rating for each rated item, turned into the most spread-out (maximum-entropy) rating distribution with that average. Like the LLM, the reference must estimate the target's ratings, which it does from items that share a feature label with it. We rescored the paper's LLM outputs against this reference. TVD is the total variation distance to this reference ×100 (0 = identical), averaged over all turns and 80 trials per model.

L-3BL-8BL-70BQ-3BQ-7BQ-14BQ-32B
Type posterior TVD ×100 (↓)51.852.746.254.448.249.247.2
Rating predictive TVD ×100 (↓)20.434.054.231.651.959.468.4
Prob. on top rating (%; reference: 31)33.242.368.743.664.071.583.9

Within each family, larger models' type beliefs move closer to the reference overall (TVD change per doubling of size: −1.4 [−2.3, −0.5] for LLaMA, −1.9 [−3.0, −0.7] for Qwen), while their rating predictions move steadily further away. The reference puts 31% on its top rating on average, and only 33% even when given the user's true type, against 69% for LLaMA-70B and 84% for Qwen-32B. This reference is now in Table 4 (App. F.1).

We also matched the information the other way: we gave both the LLM and the reference each type's average rating for the target item, so the only remaining step is to combine these with the inferred type. We ran this for Qwen 2.5 at 3B, 14B and 32B (32 trials each), and the gap persists. Type beliefs still move toward the reference with scale (−3.1 [−4.1, −2.0] per doubling), while rating predictions still move away (+9.4 [+9.1, +9.7]); Qwen-32B puts 72% on one rating, against the reference's 36%. We have added this to App. F.1.2.

We hope these additions address your concerns, and would be grateful if you would please consider increasing your score. Thank you so much for taking the time to review!

Gupta et al. (ACL 2025), "Enough Coin Flips Can Make LLMs Act Bayesian." Qiu et al. (Nature Communications, 2026), "Bayesian teaching enables probabilistic reasoning in large language models." Atwell et al., "BASIL: Bayesian Assessment of Sycophancy in LLMs."

Thanks for the detailed response, and the expanded comparisons helps to clarify my concerns. That said, I still see some overlap with prior work on Bayesian updating, inference–prediction gaps, and interaction-induced bias, and I would prefer to see a clearer novel insights beyond these existing findings. In particular, it remains unclear whether the shared formalization captures a common failure to use inferred information or different sources of error across environments. Overall I maintain my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.