acceptodds
Under review as a conference paper at ICLR 2027

How Faithfully do Language Models Acquire Probabilistic Knowledge?

Abstract

Many high-stakes decisions require calibrated distributions over plausible outcomes, not just top-ranked answers. LLMs are increasingly used to support probabilistic reasoning, yet it remains unclear how faithfully they acquire probabilistic knowledge during pretraining. We study this question in a controlled setting by generating synthetic Bayesian networks with known ground-truth distributions, verbalizing samples from these networks into text, and continually pretraining on the resulting corpora. We measure how well the models recover the underlying conditional distributions and the relations among them while varying network size, documentation policy, model scale, and training exposure. Models recover individual conditional probabilities closely, but the relationships among them exhibit what we term relational compression: models largely preserve which outcome is more probable and in which direction evidence moves a prediction, but they systematically underestimate the difference in likelihood between pairs of outcomes and how strongly evidence should shift a prediction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, probabilistic knowledge, continual pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4fxGOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a controlled benchmark of LLM probabilistic acquisition by generating synthetic Bayesian networks with known ground-truth distributions, and then verbalizing them into text for continuous pretraining. Through training over this corpus that faithfully represent these specified joint distributions, the authors show that while the LLM models the individual conditional probability closely, but the difference between different outcomes tend to be underestimated on the log-odds scale (which the authors referred to as "relation compression". Further experiment against different documentation rules and conditional updates reveal that the model tend to systematically under-express the update magnitude for strong evidences, while still undesirably being sensitive to irrelevant information.

Critical strengths
  • The paper works with a controlled setting where exact probabilistic inferences are possible, and present this in a more LLM compatible experiment design than many previous works that also probe probabilistic inference capabilities.
  • The evaluation goes beyond probability prediction accuracy, and the relational compression seems interesting.
  • documentation experiments look new and thoughtfully designed with consequential results
Critical weaknesses
  • The effect of additional independent data is unclear. It's unclear that whether additional independent examples can help beyond just more training epochs. Some sample-efficiency analysis can be helpful.
  • The experiment design and probability readout centers around one probability readout, of which the broad application can be less convincing. In particular, the controlled setting usefully isolates probabilistic learning from reporting bias, but its implications for real-world probabilistic knowledge remain unclear. Other probabilistic modeling setting might better distinguish fidelity to a training corpus from recovery of world probabilities under selective reporting.
  • The comparison experiments still do not isolate what makes probability recovery difficult. For example, it is hard to interpret the 5-variable results vs 3-variable results in Table 1. It seems that 5-variable should be the more challenging, but under different experiment settings the result does not direct compare and 5-variable ones are sometimes better.
Questions for authors
  • Will more training examples reduce the compression / large error under rare events?
  • Is there a principled explanation of the compression behavior for various attenuation / compression effect?
1 comment
Area_Chair_nPEFOpenReview ↗

Engaging with the authors

Dear reviewer,

the authors have posted a reply to your review. Please have a look and interact with the authors as needed.

Thank you, Your AC

Reviewer_h2zEOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies whether an LLM acquires well-calibrated probabilistic knowledge during pretraining. The authors train 1B and 7B language models using synthetic text corpora generated from various Bayesian networks. Specifically, the Bayesian networks are generated using fixed presets of structures, containing 3 or 5 binary nodes that indicate risk, disease, and symptoms. The conditional probability table associated with the network allows ancestral sampling. With the synthetic name ("entity tokens") of each node, each sample is verbalized into a text description, on which a language model is pretrained. The evaluation is done by comparing the probability of a target variable being equal to 1 against the corresponding ground-truth probability (which is analytically available because of binary variables + the conditional probability table).

From their experiments, the authors provide two main findings: (1) models are able to predict individual conditional probabilities, which of two events has a higher probability (ranking), or the direction of changes in probability when new evidence is provided. (2) models compress the log-odds (inverse of the sigmoid) of probability (i.e., the models underestimate the magnitude of log-odds), and the same or more severe compression occurs for the log-odds gap between two target events or the shift in log-odds after evidence.

Critical strengths
  1. The setup with synthetic Bayesian networks that enables comparison with the analytic ground-truth probability is interesting and reasonable.
Critical weaknesses

The gap between this paper and real-world LLMs.

The trained model in this study differs significantly from real-world LLMs, and the authors didn't make it clear what the study actually says about real-world LLMs. This work only considers binary variables with fixed presets of templates, which are restricted to a single domain (clinics). I agree that the framework itself is more general, but the current experiments were too restricted and too far from real-world LLMs. In my opinion, the study is more or less about whether the specific transformer model (OLMo-2) could correctly model the verbalised form of sets of binary variables. The authors should have justified why the analysis and findings in this work are generalisable to real-world LLMs, but I couldn't find any such claims except for one sentence in the conclusion (lines 475-478), which I couldn't understand. The authors didn't even discuss the potential reason for the compression behaviour, or in which case the compression could be bad in practice.

Clarity

The paper was hard for me to read. Section 3.2 is a bit confusing; it took me a very long time to understand the non-mention setup and I think the authors should add more details. Section 4 (Experiments and Results) is too wordy and contains too many details that don't directly contribute to the findings. For example, in lines 252-257, I don't understand how the results for different numbers of training epochs contribute to the findings of RQ1 (I think the results of the final checkpoint are more than enough to claim that the model struggles to predict the probabilities with more conditions). The "update magnitude" in RQ3 seems obvious because RQ1 already shows that models predict individual probabilities quite well. Lines 388-393 just repeat Section 3.2. Moreover, some of the definitions are missing, e.g., p^NM\hat{p}_{\text{NM}} and p^0\hat{p}_0.

Questions for authors
  1. I want to understand what the study tells about real-world LLMs. Do you think the findings here generalise to them?
2 comments
Area_Chair_nPEFOpenReview ↗

Engaging with the authors

Dear reviewer,

the authors have posted a reply to your review. Please have a look and interact with the authors as needed.

Thank you, Your AC

I had a misunderstanding in my previous deleted comment; sorry!

Thank you for your response.

I'm still concerned about whether the findings here are actually about LLMs. I now understand that the authors use pretrained LLMs and finetune them further on their corpus. However, we're not sure how much the finetuned model drifted from the original model. If finetuning erases the knowledge condensed in the original model, then can we call it an LLM? This should be verified (e.g., with perplexity).

I also understand that many papers have similar settings. If a paper studies a sequence model with a specific dataset and tries to generalize its findings to LLMs without sufficient justification, I honestly think that is not appropriate in science. The same applies to this paper.

I also still see no explanation of why compression occurs.

Regarding clarity, I unfortunately don't have enough time to re-read the paper. But I still think that Section 4 is too wordy to clearly present the empirical evidence for the main findings.

Since my previous rating (1) was from my misunderstanding, I raise my score to 2. I'm still leaning toward rejection because of the reasons above. These opinions might be somewhat subjective, so I'll decrease my confidence to 2.

Reviewer_z5AkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how effectively LLMs acquire probabilistic knowledge during pretraining. They verbalize observations from synthetic Bayeisan networks to create a controlled pretraining corpus, train on that corpus, and then evaluate how well the models are able to recover conditional probabilities under various conditions.

Critical strengths

A lot of prior work suggests that LLMs are not good probabilistic reasoners, and this paper tries to understand why this is the case in a controlled setting, which seems useful and important. It's also clear and easy to read.

Critical weaknesses

The evaluation is limited to one model family; ideally, it should be extended to other popular open source models like Qwen.

It seems like compression doesn't occur when you change the wording of the prompts. This is not highlighted in the narrative of the paper but is central to the results.

Questions for authors

The synthetic task is based on clinical note documentation, but the concepts are represented using custom tokens. Why not just make the task fully synthetic to further eliminate interference from concepts in pretraining?

1 comment
Area_Chair_nPEFOpenReview ↗

Engaging with the authors

Dear reviewer,

the authors have posted a reply to your review. Please have a look and interact with the authors as needed.

Thank you, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.