acceptodds
Under review as a conference paper at ICLR 2027

Exact Posterior Distillation: Teaching Language Models What To Believe And Whom To Trust

Abstract

Language model agents gather evidence from sources that may be incomplete or misleading. As evidence arrives, they need to revise both their confidence in possible answers and their trust in individual sources. We introduce BayesClue, a synthetic environment in which models infer a hidden answer from reports by reliable, noisy, and adversarial sources. The known generative process lets us compute exact posterior probabilities over answers and source types after each report. Exact Posterior Distillation (EPD) trains models to predict these probabilities. Across eight models from four families, EPD reduces belief error under changes in evidence length, candidate count, and source proportions. On a separate ten-model comparison, final-answer SFT has higher belief error than every belief-training arm, despite remaining competitive on response generation. EPD also supports answer revision beyond the training horizon and improves average performance on external evidence judgments and contextual appropriateness. These results suggest that explicit supervision of uncertainty can complement training on final answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training, posterior distillation, belief tracking, source reliability, conflicting evidence, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_enXrOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether teaching LLMs to estimate answer uncertainty and source reliability helps them handle conflicting evidence. It introduces BayesClue, a synthetic environment with reliable, noisy, and adversarial sources. Exact Posterior Distillation trains models to match exact posterior probabilities over answers and source types after each report. The paper reports improved belief prediction in this environment and gains on some external tasks involving conflicting documents and memories.

Critical strengths

S1. Exact posterior targets support controlled training and evaluation. BayesClue computes answer and source-type probabilities after each report under a specified generative model. These targets support supervision of both answer uncertainty and source reliability, and allow evaluation of how predictions change as evidence accumulates.

S2. The experiments show benefits across models and test conditions. The study covers multiple model families, longer evidence histories, larger candidate sets, and shifts in source proportions. Gains on some external benchmarks, without training on their examples, suggest some transfer beyond BayesClue.

Critical weaknesses

W1. The adversarial setting tests only a limited form of adversarial behavior. Adversarial sources have a fixed 2% truth rate and uniformly distributed errors. Once their type is inferred, their reports become useful counterevidence. The synthetic tests do not cover selective deception, reliability changes within an episode, or sources copying one another. Robustness to these forms of conflict remains unclear.

W2. The source categories and reporting accuracies need further validation. The paper varies source-type proportions but keeps reporting accuracies fixed at 95%, 65%, and 2%. It does not report sensitivity analyses for these accuracies or the number of source types. A comparison with a continuous distribution over each source's reporting accuracy could also help justify restricting reliability to three categories.

W3. Source reliability judgment is not directly evaluated on external tasks. Lower source-posterior KL supports learning within BayesClue. External evaluations mainly measure answer quality or calibration and do not isolate the role of source reliability judgment. Tests that change a source's history while keeping its current report fixed would help assess this mechanism.

W4. The effects of source order and report allocation are not isolated. Sources follow a fixed reporting order and contribute unequal numbers of reports in the default setting. The paper does not test randomized source order or balanced report counts. It remains unclear how these choices affect source reliability judgments.

Minor Comments: Although AI use is disclosed, I recommend redrawing the two main figures manually. In Figure 1, text becomes blurry when enlarged, and the background colors behind some labels differ from the surrounding areas. These issues affect readability and visual consistency.

Questions for authors

Q1. How to explain the CONFACT regressions on Qwen3.5? Table 2 reports EPD drops of 18.3 and 10.4 percentage points in macro-F1 for Qwen3.5-4B and 9B. The paper acknowledges variation across models but does not explain these regressions. Please analyze their causes and present base and post-training scores together, with variation across runs?

Q2. What is the added value of full posterior supervision? The paper acknowledges no clear average KL advantage over SAMPLE in Figure 5(b), while Figure 5(c) shows generation gains for some models. Could the authors report confidence intervals for these gains and clarify where full posterior targets offer an advantage over sampled labels? Besides, Figure 5 has overlapping elements. The y-axis title of panel (b) overlaps with data points in panel (a). Please increase the spacing between panels to improve readability.

Q3. Does source supervision still help when the supervision budget is matched? In Table 14, World-only removes source supervision and reduces the number of supervised targets. Does EPD still outperform World-only when both receive the same number of supervised targets?

5 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their feedback. Below, we address each of their concerns:

W1. Limited adversaries.

We added the harder adversary you suggest, that lies selectively. The adversary lies about two of the four features and reports the others truthfully. This breaks an assumption of our training setup, so even the exact Bayesian posterior computed under the training assumptions becomes less accurate (+.145 nats). The trained models lose a similar amount (+.07 to +.22 nats), and their predictions stay within .04 nats of that Bayesian posterior. The drop therefore comes from the deceiver violating assumptions that affect any Bayesian reader, not from the trained models breaking down. Sources whose reliability changes mid-episode or who copy each other remain future work.

W2. Reporting accuracies and number of types.

We added every variation you listed: reliable-source accuracy 0.90 and 0.85, noisy-source accuracy 0.75, adversary truth rate 0.10, and two source types instead of three. The ranking of methods holds in most comparisons. With two types, the rise in source-belief error (.16–.42 nats) tracks the rise in the best achievable error (.23–.33). Our design follows domain randomization (Tobin et al., 2017): the main arm varies the source mix per episode instead of fixing one realistic setting.

W3. Source-history test.

We ran the suggested test: we changed a source’s stated track record while keeping its current report fixed. EPD learns source reliability from whether reports agree with each other. This matches how BayesClue is built: reliability is never stated, and source types are resampled every episode, so a source’s history carries no information during training. Among all belief-training arms, EPD retains the use of stated track records best, outperforming sampled targets by .14 to .54.

W4. Order and counts.

Report order has a small effect, and training on shuffled reports removes it. Permuting reports leaves the exact posterior unchanged; under permutation, EPD’s answer-belief error rises by at most .037 nats (Qwen3.5-4B: .018 [.011, .027]), and with random sources and unbalanced counts by at most .051. On Granite-2B, training with shuffled report order removes the effect (+.046

+.024,+.069+.024, +.069

→ +.002

−.011,+.014−.011, +.014

) with no loss in-distribution.

Q1. CONFACT.

The drop is a shift toward "yes", with discrimination unchanged (AUROC −.026

−.055,+.002−.055, +.002

). Details are in our reply to 3NZ9, W2.

Q2. Value of exact targets.a~

Exact targets give more accurate answer beliefs, better generation and better revision; sampled targets are slightly better on source beliefs. Split by query type across all runs:

  • Answer beliefs: exact targets achieve lower error on every model (p<0.05)
  • Source beliefs: sampled targets are slightly better by 0.02-0.08 nats
  • Generation: exact targets beat sampled targets on both RAMDocs recall (table below)
ModelRecall: exact / sampledPrecision: exact / sampled
Qwen3.5-9B.398 ± .023 / .216 ± .041.645 ± .027 / .333 ± .054
Qwen3-4B.486 ± .061 / .230 ± .071.656 ± .047 / .324 ± .131
Qwen3.5-2B.310 ± .058 / .051 ± .057.441 ± .081 / .072 ± .089
OLMo-2-7B.304 ± .020 / .222 ± .028.548 ± .044 / .387 ± .058

Mean ± standard deviation over three training runs.

Exact targets revise answers more reliably beyond the training horizon: direction accuracy is 90.2% vs 78.8%, and new-leader selection is 64.3% vs 53.8% (Table 16). Exact targets score higher than sampled targets in all models on MMLU, ARC-Challenge and HellaSwag. We have updated Section 5.4 and Appendix M with the new results.

We have also increased the panel spacing in Figure 5.

AuthorsOpenReview ↗

Author Response Part 2

Q3. Matched world-only control.

We conduct the requested experiment and find that with matched supervision, EPD’s source-belief error is 1.2 to 4x lower than world-only training, so source targets teach a model to estimate which sources to trust. The two regimes are comparable on answer beliefs: world-only mostly matches EPD. This is expected, because each world target is already marginalized over source types, so fitting it implicitly rewards accounting for reliability. Source supervision makes that judgment explicit and measurable.
We have revised Appendix C and Table 14 with this text.

External transfer, EPD minus the matched world-only results are below:

ModelCASEBench Brier ↓CONFACT AUROC ↑
OLMo-2-1B−0.18 [−0.25, −0.12]+.035
+.007,+.060+.007, +.060

|
| OLMo-2-7B | −0.18 [−0.34, −0.03] | +.018

+.004,+.031+.004, +.031

|
| Qwen3.5-9B | −0.18 [−0.24, −0.12] | +.001

−.013,+.015−.013, +.015

|

EPD is better on most models. On RAMDocs, EPD leads the matched arm on Qwen3.5-9B by +.110

+.064,+.159+.064, +.159

in discrimination, and by +.082

+.043,+.122+.043, +.122

with every item scored in both answer orders, so the lead is not a position effect.

Figures.

We have regenerated the figures as vector graphics with high resolution, corrected spacing and consistent label backgrounds.

We hope our responses were able to address the concerns of the reviewer and look forward to further discussion and a reassessment of the scores for our work.

References

  • Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS.
Reviewer_enXrOpenReview ↗

Response to Authors' Replies

Thank the authors for the revisions and supplementary experiments, which strengthen the robustness checks and clarify the benefits of exact targets. According to your replies, I still have two questions:

1. Matched world-only control. Does EPD retain its MemBench advantage when the world-only control receives matched supervision and comparable training? Without this result, it remains unclear whether source supervision explains the memory-arbitration gain claimed in Table 14.

2. Source-history test. EPD performs below the base model on the stated source-history test, despite outperforming other belief-training arms. Does this result limit the claim that EPD improves transferable source-reliability judgments, particularly when trust depends on a source’s past performance?

I would also suggest qualifying two claims:

1. Selective deception: Keep robustness claims within the scope of the tested adversarial setting.

2. Benefits of exact targets: Acknowledge that source-belief accuracy is slightly lower than with sampled targets, and generation performance, although better than sampled targets, remains below the base model.

Reviewer_3NZ9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains LLM to track two things as evidence arrives from sources of unknown reliability: which answer is most likely, and which sources to trust. It introduces a synthetic detective game environment (BayesClue) to train the LLMs with in total 216 joint states. Besides, the paper proposes Exact Posterior Distillation (EPD) method to train LLMs by minimizing KL divergence loss over answer labels. The results are tested both on the simulated environment and OOD external benchmarks, using various metrics to demonstrate the improvement. However, the setting of the simulated environment is too naive to mimic the real world scenario, though the authors claim that models trained on it can lead to OOD improvement.

Critical strengths
  1. The paper focus on a very important question that source of LLM inputs may not be reliable. The idea of training inside a simulated environment with exact posteriors that we can't get from real data is somewhat interesting.
  2. The paper is well written and logically consistent, with simple method and clear motivation.
  3. The experiments are comprehensive across diverse OOD tasks, although the analysis can be better improved.
Critical weaknesses
  1. The simulated environment is too simplistic for real-world scenarios, and its assumptions are not validated. The authors make several strong design choices without justification or sensitivity analysis, e.g. only 216 joints states and fixed reporting accuries (95%, 65%, 2%), 20% of sources are adversarial (this is counterintuitive as most sources should be helpful). Therefore, it's unclear how results change if these choices change, for example lowering reliable-source accuracy from 95% to 90%, or using two source types instead of three. Due to the reason, all experiments on the simulated environment becomes less convincing, which leaves the OOD benchmark evaluation the only reliable source. I'm wondering if the authors ground such strong hypothesis stated in weakness 1 in empirical studies of real source reliability, or the designs are just based on their own intuition?
  2. The experiments are weakly explained, and the mechanism behind the gains is unclear. The paper reports improvements on many benchmarks, but it does not explain why EPD helps or analyze how the gains arise on each one. For example, CONFACT improves sharply on small Gemma and OLMo models but drops on the larger Qwen3.5 models, and it is unclear whether this reflects better weighing of conflicting evidence or simply a corrected yes/no bias in weak base models. Similarly, the models with the best internal belief accuracy show little or negative transfer, RAMDocs gains start from near-chance baselines, and on CASEBench the fixed-mix and sampled-label variants often outperform EPD itself. Without per-benchmark analysis of which items improve, how confidence and label preferences shift, and whether gains concentrate where source reliability matters, the transfer cannot be attributed to learning whom to trust rather than to a generic calibration effect of the shared single-token answer format.
  3. The baselines and evaluation setting are too weak to be convincing. The only prompting baseline is a single instruction to consider source reliability, applied to models that mostly answer with one forced token, which leaves no room for reasoning. This is far from how LLMs are deployed in practice, where stronger models are typically used with chain-of-thought, reasoning modes or agentic harnesses that may already reason about which sources to trust. The authors should show that EPD still adds value in such practical settings, for example when integrated into a reasoning pipeline, even if the gains are modest.
Questions for authors

please refer to the weaknesses

4 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their thoughtful feedback. Below, we address each concern:

W1: Simplicity of the environment.

We would like to clarify two things:

  1. The 20% adversarial share applies only to the fixed-mix control arm. The main arm draws a new source mix for every episode from Dirichlet(1,1,1), so training spans mostly-reliable, mostly-noisy, and adversarial-heavy populations.
  2. The reporting rates are design parameters, instead of being estimates of real-world reliability. This follows domain randomization (Tobin et al., 2017): instead of matching one realistic setting, we vary the simulator enough that the real world becomes one more variation. The same principle underlies transfer from procedurally generated environments in LLMs. About 5K synthetic puzzles with truth-telling and lying agents improved a 7B model on AIME and AMC (Xie et al., 2025). Synthetic logic corpora improve math, coding, and BBH (Morishita et al., 2024; Liu et al., 2025). We include adversarial sources because unreliable context is a documented failure mode: LLMs adopt incorrect retrieved content over their own correct knowledge more than 60% of the time (Wu et al., 2024).

As requested, we have tested the trained models on nine held-out conditions that each change one assumption. These include:

  • reliable-source accuracy 0.90 and 0.85,
  • noisy-source accuracy 0.75,
  • adversary truth rate 0.10,
  • two source types,
  • shuffled report order,
  • random sources and
  • selective deception.

The ranking of methods and conclusions of our paper still hold across all settings. Where error does rise, it’s mostly because the task itself gets harder. For example, with only two source types, even an ideal predictor does worse, and our models degrade by a similar amount. Full results are in Appendix N.

We use 216 states by default because we can compute the exact training targets cheaply, the method is not limited to it. We already test on larger candidate sets and on histories eight times longer than those seen in training (Figures 3–4). Finally, all external evaluations use benchmarks never seen in training, which tests whether the learned behavior transfers beyond the simulator’s assumptions.

W2. Mechanism of external gains.

We ran the requested diagnostics and now attribute each gain:

  • CONFACT: The Qwen3.5 drop is not worse evidence weighing but a shift toward answering “yes”. Discrimination is unchanged, with AUROC −.026
−.055,+.002−.055, +.002

on Qwen3.5-4B and −.017

−.042,+.006−.042, +.006

on Qwen3.5-9B. On Granite-2B and OLMo-2-7B, AUROC rises by .15 and .11, so the gains are real discrimination performance improvements instead of a bias.

  • RAMDocs: The gain is calibrated refusal of planted misinformation. It is the same whether or not a majority vote already finds the answer (difference +.001
−.125,+.125−.125, +.125

), so it is not majority counting. With answer positions balanced, it remains significant.

  • Answer-format artifact: Every uncertainty training regime lowers mean CASEBench Brier relative to the base (13.60): EPD 9.59, SAMPLE 8.44, FIXED 8.56, and world-only 11.08. MAP, which uses the identical single-token format but keeps only the leading label, is worse than the base (15.88). The gain therefore comes from learning to preserve uncertainty, not from the shared answer format.
  • Variation across models: We explicitly state in the paper that transfer magnitude does not track internal belief error across models.
AuthorsOpenReview ↗

Author Response Part 2

W3. Reasoning baselines.

We added a reasoning-mode comparison on four Qwen models. EPD’s gains come in direct answering, at no additional inference cost. It raises RAMDocs discrimination by +.102

+.030,+.170+.030, +.170

on Qwen3.5-4B and +.189

+.107,+.274+.107, +.274

on Qwen3.5-2B.

The table reports results on RAMDocs conflict items, in thinking mode, up to 16,384 thinking tokens. Metric: gold picks minus misinformation picks.

ModelBase, directEPD, directBase, reasoningEPD, reasoningEPD − base, reasoning
Qwen3.5-2B.132.314.292.364+.071
−.019,+.158−.019, +.158

|
| Qwen3.5-4B | .457 | .564 | .683 | .668 | −.015

−.080,+.051−.080, +.051

|
| Qwen3.5-9B | .527 | .492 | .498 | .674 | +.176

+.110,+.241+.110, +.241

|
| Qwen3-4B | .469 | .451 | .519 | .591 | +.073

+.018,+.129+.018, +.129

|

On Qwen3.5-9B EPD completes answers with 10–25% fewer thinking tokens. With zero thinking tokens, EPD outperforms the reasoning base model performance.
EPD completes its reasoning within budget far more often (44–54% vs. 18.5%, on average). EPD therefore improves the low-cost direct-answer mode and is complementary to reasoning pipelines.

We have also added stronger baselines by comparing with decision models like Jev, GPT Luna Decisions and Laya. Most decision models do not track whom to trust. On revision beyond the training horizon, EPD performs the best:

ModelCorrect direction (%) ↑New leader (%) ↑
Qwen3.5-4B base48.912.0
Laya53.514
Luna6137
Jev67.548.5
EPD (Qwen3.5-4B)90.264.3

We hope our responses were able to address the concerns of the reviewer and look forward to further discussion and a reassessment of the scores for our work.

References

  • Liu, J., Fan, Y., Jiang, Z., et al. (2025). SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond. arXiv:2505.19641.
  • Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2024). Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus. NeurIPS.
  • Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS.
  • Wu, K., Wu, E., & Zou, J. (2024). ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. NeurIPS.
  • Xie, T., Gao, Z., Ren, Q., et al. (2025). Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning. arXiv:2502.14768.

Thank you for your reply and additional experiments. W3 has been fully addressed, but my concerns regarding W1 and W2 remain. For W1, varying the simulator parameters does not fully address the gap between the synthetic environment and real-world scenarios, where source reliability may change across topics and over time. For W2, I'm still not convinced that the external gains come from learning source reliability rather than simply learning to better express uncertainty. Overall, my concerns have been partially addressed, I choose to keep my original rating, while increasing the Informativeness score.

Thank you for the follow-up and for raising the informativeness score. We’re glad the reasoning-mode results addressed W3.

W1. We would like to note that BayesClue is meant to act as a training environment, following prior works and not to be fully realistic. Its role is to provide exact training targets (which realistic settings do not contain), and real-world relevance is tested on the external benchmarks, which contain real conflicting sources and where EPD improves average performance. We state this as a limitation in Section 6 and see environments with topic- and time-dependent reliability as the natural next step.

W2. We would gently question the distinction between the two explanations. In BayesClue, expressing uncertainty correctly already requires accounting for source reliability: each answer target is averaged over which sources might be reliable or adversarial. This is why world-only training matches EPD on answer beliefs. Our matched control therefore shows that explicit source targets add little on external tasks, not that reliability plays no role. That said, we have revised the claim to state that EPD improves uncertainty-aware answer selection on external tasks, while source-reliability judgments are learned and measured within BayesClue.

We hope our clarifications were able to address your remaining concerns and look forward to engage more with you!

Reviewer_X7r3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BayesClue, a synthetic environment in which models infer a hidden answer from sequential reports by reliable, noisy, and adversarial sources. The known generative process enables exact computation of posterior probabilities over answers and source types. Exact Posterior Distillation (EPD) trains language models to predict these distributions using a KL-divergence objective. Evaluations across multiple model families show improved posterior fidelity under synthetic distribution shifts and some transfer to external evidence-judgment and context-sensitive tasks. Gains in calibration and answer generation vary across models, and exact targets do not consistently outperform posterior-sampled supervision.

Critical strengths
  • BayesClue enables controlled study of belief updates using exact posterior targets over answers and source reliability.
  • Evaluation covers multiple model families and sizes up to 9B, with repeated runs and substantial posterior-fidelity gains.
  • External tasks show encouraging transfer without task-specific training, including improved evidence judgments and average Brier scores.
Critical weaknesses
  • W1: The framing does not sufficiently distinguish EPD from prior Bayesian belief training and explicit belief distillation. Cui and Lin (2026), already study explicit posterior-belief distillation but are not discussed, leaving EPD’s specific advance unclear.
  • W2: The results support belief supervision more clearly than the need for full exact-posterior targets. Exact targets show no clear average KL advantage over sampled targets, while generation and calibration advantages are model-dependent.
  • W3: The paper does not establish whether training solely on the posterior objective preserves general-purpose language capabilities, limiting its assessment as a broader post-training method.
  • W4: The broader applicability of exact-posterior supervision remains unclear because obtaining its targets requires a known, tractable generative process. External evaluations show encouraging behavioral transfer, but do not establish accurate posterior estimation on substantially different problems.
Questions for authors
  1. How does EPD differ from BOND (Cui and Lin, 2026), and how would a matched belief-distillation baseline compare?
  2. Given the similar KL results for exact and sampled targets, what reliable downstream advantage does full-posterior supervision provide?
  3. Does EPD-only training degrade unrelated language capabilities? Have you tested combining it with a capability-preserving objective?
  4. Can you clarify when exact posterior targets are practical to obtain beyond BayesClue, and what evidence supports posterior-estimation transfer to a different problem?
  5. Would source-reliability supervision still help if the world-only control received an equal amount of supervision?
  6. Can you report absolute generation scores for exact versus sampled targets, including precision alongside recall and uncertainty across runs, to clarify the practical gains?
3 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their insightful comments and feedback. Below we address each concern:

W1/Q1. Comparison with BOND

We have added the positioning with BOND in the related work: BOND distills approximate beliefs estimated by an LLM, while EPD distills exact posteriors over both answers and source reliability. BOND does not model source reliability, and its own recursive Bayesian variant scores are worse than a uniform belief (Brier 0.176 vs 0.139; their Table 3).

W2/Q2/Q6. Value of exact targets.

Exact targets give more accurate answer beliefs, better generation and better revision; sampled targets are slightly better on source beliefs. Split by query type across all runs:

  • Answer beliefs: exact targets achieve lower error on every model (p<0.05)
  • Source beliefs: sampled targets are slightly better by 0.02-0.08 nats
  • Generation: exact targets beat sampled targets on both RAMDocs recall (table below)
ModelRecall: exact / sampledPrecision: exact / sampled
Qwen3.5-9B.398 ± .023 / .216 ± .041.645 ± .027 / .333 ± .054
Qwen3-4B.486 ± .061 / .230 ± .071.656 ± .047 / .324 ± .131
Qwen3.5-2B.310 ± .058 / .051 ± .057.441 ± .081 / .072 ± .089
OLMo-2-7B.304 ± .020 / .222 ± .028.548 ± .044 / .387 ± .058

Mean ± standard deviation over three training runs.

Exact targets revise answers more reliably beyond the training horizon: direction accuracy is 90.2% vs 78.8%, and new-leader selection is 64.3% vs 53.8% (Table 16). Exact targets score higher than sampled targets in all models on MMLU, ARC-Challenge and HellaSwag. We have updated Section 5.4 and Appendix M with the new results.

W3/Q3. Capability retention.

We added standard capability benchmarks, and EPD largely preserves knowledge and reasoning. On Qwen3.5-4B (three runs, 95% CIs), EPD changes MMLU by +2.8 [1.0, 4.6], ARC-Challenge by +2.6 [0.8, 4.4] and HellaSwag by −2.0 [−2.9, −1.2]. On Qwen3-4B the changes are −2.0, +2.4 and +0.9. Sampled targets lose accuracy on all three benchmarks. Wikitext-2 perplexity on Qwen3.5-4B is within 1% of the base under EPD; sampled targets add 20–25%. Against the budget-matched world-only regime on Qwen3.5-4B, EPD is higher on MMLU, ARC-Challenge and HellaSwag (+1.9, +2.6, +0.6).

Results are in new Appendix M (Table 25) and noted in our contributions (Sec. 1) in the updated draft.

W4/Q4. Applicability.

We use exact posteriors only as a training curriculum. The trained model is applied to new tasks without any generative model, so the requirement is that one tractable training environment exists. This is the recipe behind prior-fitted networks (Müller et al., 2022; Hollmann et al., 2025) and Bayesian teaching (Qiu et al., 2026). Such environments are cheap to scale. Given the candidate answer, sources are independent, so exact inference grows linearly with the number of sources rather than exponentially.

Since real tasks have no ground-truth posteriors, BayesClue provides exact posteriors to help in training. There, we measure whether the model’s probabilities become more informative:

  • On MemBench, EPD’s confidence ranks its own correctness better: Accuracy on its most confident 80% of answers rises from .638 to .720 (Qwen3.5-4B).
  • On CONFACT, AUROC rises by .15 and .11 on Granite-2B and OLMo-2-7B.
  • Within BayesClue, posteriors stay accurate on histories 8× longer than any seen in training (world KL .18 at 64 reports, against 2.7 for the base). Section 6 states that the external results show transferred probability quality, not exact posterior estimation on real tasks.

Q5. Budget-matched world-only control.

We conduct the requested experiment and find that with matched supervision, EPD’s source-belief error is 1.2 to 4x lower than world-only training, so source targets teach a model to estimate which sources to trust. The two regimes are comparable on answer beliefs: world-only mostly matches EPD. This is expected, because each world target is already marginalized over source types, so fitting it implicitly rewards accounting for reliability. Source supervision makes that judgment explicit and measurable.
We have revised Appendix C and Table 14 with this text.

The source-belief gap widens under a reversed source mix (Qwen3.5-9B: from 1.3–2.0× to 1.8–2.5×). External transfer, EPD minus the matched world-only results are below:

ModelCASEBench Brier ↓CONFACT AUROC ↑
OLMo-2-1B−0.18 [−0.25, −0.12]+.035
+.007,+.060+.007, +.060

|
| OLMo-2-7B | −0.18 [−0.34, −0.03] | +.018

+.004,+.031+.004, +.031

|
| Qwen3.5-9B | −0.18 [−0.24, −0.12] | +.001

−.013,+.015−.013, +.015

|

EPD is better on most models. On RAMDocs, EPD leads the matched arm on Qwen3.5-9B by +.110

+.064,+.159+.064, +.159

in discrimination, and by +.082

+.043,+.122+.043, +.122

with every item scored in both answer orders, so the lead is not a position effect.

AuthorsOpenReview ↗

Author Response Part 2

We hope our responses were able to address the concerns of the reviewer and look forward to further discussion and a reassessment of the scores for our work.

References

  • Cui, Z. & Lin, B. (2026). Distilling Bayesian Belief States into Language Models for Auditable Negotiation. arXiv:2605.04507.
  • Hollmann, N., Müller, S., Purucker, L., et al. (2025). Accurate predictions on small data with a tabular foundation model. Nature, 637, 319–326.
  • Müller, S., Hollmann, N., Pineda Arango, S., Grabocka, J., & Hutter, F. (2022). Transformers Can Do Bayesian Inference. ICLR.
  • Qiu, L., Sha, F., Allen, K., Kim, Y., Linzen, T., & van Steenkiste, S. (2026). Bayesian teaching enables probabilistic reasoning in large language models. Nature Communications, 17, 1238.
AuthorsOpenReview ↗

Gentle Reminder

Dear reviewer X7r3, we deeply appreciate the time and effort you invested in reviewing our paper. Since the discussion period closes soon, we hope our response and additional experiments address all your concerns appropriately and that you will reconsider the score accordingly.

We look forward to further discussion on any questions or comments you still have. Please let us know.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Author Response

We thank all reviewers and the AC for their time and engagement during the discussion.

Contributions. We introduce BayesClue, a synthetic environment that gives exact posterior targets over both answers and source reliability after every report, and Exact Posterior Distillation (EPD), which trains LLMs on these targets. Across eight models from four families, EPD reduces belief error under longer histories, larger candidate sets and shifted source mixes, supports answer revision beyond the training horizon, and improves average performance on external evidence-judgment benchmarks without training on them.

Strengths noted by reviewers. Exact targets enable a controlled study of belief updating (X7r3, enXr). The problem is important and the paper is clear and well motivated (3NZ9). The evaluation is broad, with multiple model families and repeated runs (X7r3, enXr), and the experiments are solid (3NZ9).

Added during the discussion:

• Capability retention on MMLU, ARC-Challenge and HellaSwag

• A budget-matched world-only control, including the MemBench breakdown

• Exact versus sampled targets, split by query type, with generation and revision results

• Nine held-out generators that each change one assumption (805/810 method rankings preserved)

• Analyses of what drives the gains on CONFACT, RAMDocs and CASEBench

• A reasoning-mode comparison and the source-history test

• A new environment with topic-dependent and time-varying source reliability, trained with the unchanged EPD recipe

Following the reviewers’ suggestions, we reframed our claims: external gains reflect better uncertainty-aware answer selection, while source-reliability judgments are learned and measured within BayesClue. enXr raised their score after these revisions.

Reviewer X7r3 has not responded to our rebuttal. Our replies address each of their points: the BOND comparison, the value of exact targets, capability retention, applicability, and the budget-matched control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.