- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how LLMs trained on Bayesian data through SFT represents and uses those learnings internally. The previous paper on this subject established the performance gains from trained LLMs on Bayesian data, while this paper provides additional understanding into the mechanics of those LLM learnings. Specifically, this paper studies whether some of the resulting LLM layers' hidden representations can recover prior, posterior, and likelihood updates; whether those representations casually influence the final recommendations; and whether the model uses those internal representations in a Bayesian manner.
Critical strengths
The paper sheds more light on how LLMs learn based on Bayesian data. It conducts several interesting experiments, particularly the intervention experiment in Section 5, which investigates if some internal activations in layers where the LLM starts learning the Bayesian quantities better casually impact the final recommendation.
Critical weaknesses
Motivation and practicality
Overall, the results are not very surprising given that the model-at-issue was trained on Bayesian data. So the fact that the LLMs can partially recover the quantities used by Bayes’ rule is not very surprising.
It’s unclear how the current insights are practically useful. For example, Section 4 experiments show that BayesLLM representations further trained through some linear probes to output Bayesian-ground truth quantities can recover those quantities better than the OracleLLM. But these experiments do not show that the LLM itself without further training can already output those quantities. An interesting extension would be if the BayesLLM can also output some Bayesian quantities such as posterior variances, Bayesian credible intervals, or some other meaningful quantities of the Bayesian posterior rather than just the flight recommendations. For example, could the model output the recommendations together with uncertainties?
Experimental Limitations
The paper uses a single dataset, which is very simple; with discrete preferences and the final utility depends only on the posterior mean.
Some experiments are not done across all models, and statistical significance analyses are missing for key experiments.
Questions for authors
See the weaknesses plus these additional questions.
-
Are the Bayesian accuracy R1 results Figure 21 statistically significant for every model and test set? The CI results, especially for Qwen model, overlap, so some of them may not be statistically significant. It’s stronger to provide this analysis for every model and test set separately.
-
Same for the Oracle accuracy R1 results. Are the differences across models and test sets statistically significant?
-
Can you provide the full analysis as in Figure 4 including prior, posterior, and likelihood for other models? I see only the prior analysis in the Appendix.
-
The balanced accuracy R3 Section 5 results seem much weaker on Qwen model. Any explanation for that? Can the author report confidence intervals for balanced accuracy (intervention experiment in Section 5) and for the differences between models?
-
Can the author include the results of the second Experiment for Section 5 for the other models as well as the significance levels?
2 comments
Follow-up to Reviewer Miou
Dear Reviewer Miou,
As the author response period ends soon, we would be grateful for your feedback on our rebuttal. Please let us know if any concerns remain that we can still address.
Also, while responding to the other reviewers, we revised the manuscript, and this changed some table numbers. We have updated our response to you so its table references match the current version.
Thank you again for your time and constructive comments.
Additional analyses, results across backbones and additional tasks
We thank the reviewer for the helpful feedback.
A key contribution of the paper is the framework for evaluating whether language models are Bayesian decision makers, rather than only the conclusion about whether a particular LM is Bayesian.Therefore, the specific results on a given task becomes less central, since we can argue that the same framework could be applied to other models even if they lead to different conclusions. Nonetheless, to address the experimental limitations, we now additionally report results on two additional tasks - a hotel recommendation task (Qiu et al., 2026), and a baseline nonsense task (added in Appendix I). While the hotel task tests the model’s generalization to a different domain, experiments on nonsense data allow to remove any semantic interference from possible in-weight world knowledge, and isolate the probabilistic reasoning task. It uses the same structure as the flight recommendation task, but uses nonsense feature names and simply calls the options “items”. Crucially, the key insights gained through our framework on the flight recommendation task are further strengthened by these additional analyses. The analyses on both tasks confirm that BayesLM outperforms the OracleLM behaviorally, although the gap decreased. The models also performed slightly worse on the nonsense task. Interestingly, probing results confirm that BayesLM learns to represent more accurate beliefs than OracleLM on both tasks, although the models haven’t been trained on these tasks. Second, the neuro-symbolic analysis shows a more substantial gap than on the flight recommendation task between both models’ performance and the neuro-symbolic accuracy, suggesting that the models may generalize well to representing probabilistic beliefs on new tasks (the neuro-symbolic accuracy comes close to BayesLM’s original flight accuracy), but that the reading of these beliefs for decision making might be sensitive to the fine-tuning domain. We acknowledge that these additional tasks still rely of a similar structure, and discuss these limitations and directions for future work in Appendix A.
We address further feedback and questions point by point below.
Motivation & practicality: We thank the reviewer for these questions. In response, we add a set of evaluations in Appendix D (‘verbalization of beliefs’), which elicit the LMs’ beliefs about the user’s preferences through prompting (‘verbalization’). Similar analyses were conducted by Qiu et al. (2026), who show that BayesLM can better verbalize its beliefs than OracleLM or the untrained LMs. However, these analyses haven’t directly assessed the calibration of the verbalized beliefs. We compare beliefs accessed through verbalization to the model encoded information that we assess through probing. The verbalized beliefs show a considerably larger divergence than the probed beliefs, suggesting that verbalization does not fully retrieve the information the LM encodes. This suggests that Bayesian fine-tuning leads to more accurate Bayesian behavior and beliefs than standard fine-tuning, but still doesn't guarantee that verbalized beliefs are calibrated with respect to the models' encoded information. Full details are now reported in Appendix D.
Experimental limitations: We thank the reviewer for pointing this out, and now completed the statistical analyses and added remaining results for all backbones in the Appendices B, F, H.
Answers to the questions: