- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how LLMs update their beliefs during multi-turn interactions where evidence arrives sequentially. It introduces BayesBench, a evaluation suite structured around three core Bayesian tasks instantiated in four distinct environments: (1) Bayesian estimation via a coin-flip task with unknown bias; (2) Bayesian prediction via a cold-start recommender system predicting held-out movie ratings from sequential ratings; and (3) latent-framed Bayesian prediction via social judgment (r/AmItheAsshole) and medical triage tasks, where observations are filtered through a user's communication style (latent persona).
Critical strengths
Rigorously Controlled Multi-Turn Evaluation Framework: BayesBench moves beyond standard single-turn final-answer evaluations by tracking intermediate belief trajectories across sequential turns. The integration of closed-form Bayesian references (for estimation and recommendation tasks) alongside debiased multi-choice option probing via Latin squares provides a methodologically sound and mathematically grounded way to isolate belief-updating dynamics from simple output formatting artifacts.
Identification of the "Inference-Prediction Gap": The paper uncovers a valuable and counter-intuitive insight: while model scaling reliably improves latent variable recovery (e.g., identifying user types or patient communication styles), this capability does not automatically yield better-calibrated downstream predictions. This exposes a critical limitation in current LLM architectures and post-training methods regarding how implicit latent information is operationally utilized.
Critical weaknesses
Reliance on Constrained and Discrete Latent Structures: The simulation environments heavily rely on simplified, highly discretized categorical spaces (e.g., movie-viewer profiles or fixed communication styles). Real-world user interactions feature continuous, mixed, and dynamically shifting preferences that overlap in complex ways, potentially limiting how well these findings generalize to organic human-chatbot dialogues.
Confounding Effects of LLM-Simulated Interlocutors: In the active engagement tasks (social judgment and medical triage), user dynamics are generated by LLM simulators following fixed style prompts. This introduces potential artifacts or biases inherent to the simulator models themselves (such as predictable framing styles or stylized lexical choices), which may not fully capture the chaotic and irregular feedback loops of true human interlocutors.
Inherent Limitations of Multiple-Choice Probing: Belief elicitation relies on extracting token log-probabilities over closed-form multiple-choice options (via Latin square counterbalancing). While this controls for position bias, it forces a discrete format that may restrict the model's expressive capacity and fail to completely mirror internal confidence or beliefs that would otherwise emerge in free-form, unconstrained text generations.
Questions for authors
Clarification Request: In the latent-framed tasks (social judgment and medical triage), explicit conditioning on the model's argmax inferred latent improved downstream prediction accuracy for certain model scales (e.g., Figures 4 and 7). Given that explicit prompting bridges part of the gap, have you experimented with reinforcement learning or fine-tuning techniques (such as training on intermediate Bayesian reasoning traces) to close the gap natively without requiring manual latent injection at inference time?
Suggestion for Future Work: It would be valuable to understand how proprietary frontier models (e.g., GPT-4 or Claude 3.5 Sonnet) perform on BayesBench compared to the open-weight models evaluated here, particularly regarding whether scale alone in massive closed models mitigates extreme over-updating or if it represents a fundamental limitation of next-token prediction architectures.
2 comments
W3. "…it forces a discrete format that may restrict the model's expressive capacity and fail to completely mirror internal confidence or beliefs that would otherwise emerge in free-form, unconstrained text generations."
A3. Fixed-choice probing gives a full distribution over a fixed set of outcomes, which we need in order to compare with the Bayesian reference at every turn. It also avoids splitting probability across different phrasings of the same answer, a limitation Gupta et al. note; fully free-form answers would have to be mapped back to the outcomes, which reintroduces this problem. To test whether the result depends on the probe, we had the models state a percentage for each option in words, on the same conversations (response to Reviewer cacN, W2). The two readouts differ, but under both, type beliefs improve with scale and rating predictions do not move toward the reference; the stated ratings are broad but barely respond to the evidence. We keep token probabilities as the primary readout because they are the distribution the model's answers are generated from; Pal et al. (2026) find that elicited confidences often fail to predict actions.
Q1. "…have you experimented with reinforcement learning or fine-tuning techniques … to close the gap natively without requiring manual latent injection at inference time?"
A4. We have not; training is out of scope for this work, but it is a very interesting direction. Qiu et al. show that fine-tuning on a Bayesian assistant improves interactive recommendations, so training toward Bayesian behavior is feasible; our results suggest targeting the step from inferred latent to prediction. As you note, latent-conditioned prediction shows that some models can use the inferred latent when it is written into the prompt, though the gains are modest and uneven; training would aim for the same without that injection. BayesBench measures this directly: training succeeds to the extent that the prediction without the injected latent approaches the prediction with it. To keep the benchmark an evaluation, training would use separately generated environments rather than the benchmark's own.
Q2. "It would be valuable to understand how proprietary frontier models … perform on BayesBench compared to the open-weight models evaluated here…"
A5. Our probe needs token log-probabilities for the answer options, which current frontier APIs return only for some models and settings (for example, not for reasoning models), as Gupta et al. also note. Asking models to state probabilities in words would avoid this. In our test of that (response to Reviewer cacN, W2), stated beliefs about the user's type responded to the evidence, but stated rating predictions stayed nearly the same whatever the evidence. Stated probabilities therefore cannot yet replace token probabilities for rating prediction, and evaluating closed models needs a better way to elicit those predictions, which we list as future work.
We hope these additions, including the test of the fixed-choice probe against stated probabilities (A3), address your concerns, and would be grateful if you would please consider raising your score. Thanks so much!
Gupta et al. (ACL 2025), "Enough Coin Flips Can Make LLMs Act Bayesian." Qiu et al. (Nature Communications, 2026), "Bayesian teaching enables probabilistic reasoning in large language models." Pal et al. (2026), "Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions."
We thank the reviewer for describing the evaluation as rigorously controlled and the inference–prediction gap as a valuable and counter-intuitive finding. We have added a Bayesian reference given exactly the LLM's information (response to Reviewer fhBu, Q1), and credible intervals for all main claims. Against this reference, larger models still infer the user's type better while their rating predictions move further away, and the credible intervals for both trends exclude zero. We address each point below.
W1. "The simulation environments heavily rely on simplified, highly discretized categorical spaces … potentially limiting how well these findings generalize to organic human-chatbot dialogues."
A1. We agree that organic dialogues are richer than our environments, and our limitations discuss this (App. B). The structure is deliberate: scoring the model's belief at every turn needs a known likelihood, so that a Bayesian reference, or conditions any Bayesian reasoner must satisfy, can be computed. Within this constraint, the design is not limited to categories: The generative process in Sec. 2.1 places no restriction on the latent, which is continuous in Task 1 (the coin's bias) and categorical in Tasks 2–3, and the same template allows continuous or mixed latents, which we list as future work. The environments also move toward organic dialogue step by step. The recommender's user types are fitted to real MovieLens ratings, and social judgment is built on real r/AmItheAsshole posts; in social judgment and medical triage, each case is delivered over many turns in different communication styles, and we test conditions any Bayesian reasoner must satisfy instead of an exact posterior (response to Reviewer fhBu, W3). The inference–prediction gap appears even in the controlled recommender, where the reference is exact.
W2. "…user dynamics are generated by LLM simulators following fixed style prompts. This introduces potential artifacts or biases inherent to the simulator models themselves…"
A2. We agree, and our limitations discuss this (App. B). In terms of Eq. (1), the simulator implements only the framing of each observation, ℓηf(xt∣x~t,zf), which limits what it can introduce: