acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow

57 views

Abstract

Language models (LMs) are increasingly used for tasks that require reasoning about hidden variables from a few observations, for which Bayesian inference is the normatively correct solution. While supervised fine-tuning of an LM on the outputs of an optimal *Bayesian* model leads to near-Bayesian behavior, standard supervised fine-tuning (SFT) on the true answers for the task falls short of it. But behavior alone does not tell us *why* tuning on a *Bayesian* or an *oracle* (true answers) signal differs: whether the resulting LM represents Bayesian beliefs, acts on them, or turns them into a choice the way Bayes' rule does. To compare them, we formulate increasingly demanding requirements for an LM to count as a Bayesian decision maker, spanning its behavior, representations, and computations, and test them on a flight recommendation task. The Bayes-trained LM acts Bayesian, encodes quantities of Bayes' rule in its middle layers, and uses the encoded belief for the recommendation to a certain extent. The oracle-trained LM differs from it both in the beliefs it holds and whether it reads beliefs out into recommendations. Exchanging beliefs between the LMs transfers a part of the Bayesian advantage. Bayes fine-tuning thus installs usable Bayesian beliefs in an LM for reasoning under uncertainty in a way standard SFT on oracle answers cannot, highlighting the advantage of nuanced supervision.

Primary area:
interpretability and explainable AI
Keywords:
Language models, Bayesian inference, interpretability, belief representation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_MiouOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how LLMs trained on Bayesian data through SFT represents and uses those learnings internally. The previous paper on this subject established the performance gains from trained LLMs on Bayesian data, while this paper provides additional understanding into the mechanics of those LLM learnings. Specifically, this paper studies whether some of the resulting LLM layers' hidden representations can recover prior, posterior, and likelihood updates; whether those representations casually influence the final recommendations; and whether the model uses those internal representations in a Bayesian manner.

Critical strengths

The paper sheds more light on how LLMs learn based on Bayesian data. It conducts several interesting experiments, particularly the intervention experiment in Section 5, which investigates if some internal activations in layers where the LLM starts learning the Bayesian quantities better casually impact the final recommendation.

Critical weaknesses

Motivation and practicality

Overall, the results are not very surprising given that the model-at-issue was trained on Bayesian data. So the fact that the LLMs can partially recover the quantities used by Bayes’ rule is not very surprising.

It’s unclear how the current insights are practically useful. For example, Section 4 experiments show that BayesLLM representations further trained through some linear probes to output Bayesian-ground truth quantities can recover those quantities better than the OracleLLM. But these experiments do not show that the LLM itself without further training can already output those quantities. An interesting extension would be if the BayesLLM can also output some Bayesian quantities such as posterior variances, Bayesian credible intervals, or some other meaningful quantities of the Bayesian posterior rather than just the flight recommendations. For example, could the model output the recommendations together with uncertainties?

Experimental Limitations

The paper uses a single dataset, which is very simple; with discrete preferences and the final utility depends only on the posterior mean.

Some experiments are not done across all models, and statistical significance analyses are missing for key experiments.

Questions for authors

See the weaknesses plus these additional questions.

  1. Are the Bayesian accuracy R1 results Figure 21 statistically significant for every model and test set? The CI results, especially for Qwen model, overlap, so some of them may not be statistically significant. It’s stronger to provide this analysis for every model and test set separately.

  2. Same for the Oracle accuracy R1 results. Are the differences across models and test sets statistically significant?

  3. Can you provide the full analysis as in Figure 4 including prior, posterior, and likelihood for other models? I see only the prior analysis in the Appendix.

  4. The balanced accuracy R3 Section 5 results seem much weaker on Qwen model. Any explanation for that? Can the author report confidence intervals for balanced accuracy (intervention experiment in Section 5) and for the differences between models?

  5. Can the author include the results of the second Experiment for Section 5 for the other models as well as the significance levels?

2 comments
AuthorsOpenReview ↗

Additional analyses, results across backbones and additional tasks

We thank the reviewer for the helpful feedback.

A key contribution of the paper is the framework for evaluating whether language models are Bayesian decision makers, rather than only the conclusion about whether a particular LM is Bayesian.Therefore, the specific results on a given task becomes less central, since we can argue that the same framework could be applied to other models even if they lead to different conclusions. Nonetheless, to address the experimental limitations, we now additionally report results on two additional tasks - a hotel recommendation task (Qiu et al., 2026), and a baseline nonsense task (added in Appendix I). While the hotel task tests the model’s generalization to a different domain, experiments on nonsense data allow to remove any semantic interference from possible in-weight world knowledge, and isolate the probabilistic reasoning task. It uses the same structure as the flight recommendation task, but uses nonsense feature names and simply calls the options “items”. Crucially, the key insights gained through our framework on the flight recommendation task are further strengthened by these additional analyses. The analyses on both tasks confirm that BayesLM outperforms the OracleLM behaviorally, although the gap decreased. The models also performed slightly worse on the nonsense task. Interestingly, probing results confirm that BayesLM learns to represent more accurate beliefs than OracleLM on both tasks, although the models haven’t been trained on these tasks. Second, the neuro-symbolic analysis shows a more substantial gap than on the flight recommendation task between both models’ performance and the neuro-symbolic accuracy, suggesting that the models may generalize well to representing probabilistic beliefs on new tasks (the neuro-symbolic accuracy comes close to BayesLM’s original flight accuracy), but that the reading of these beliefs for decision making might be sensitive to the fine-tuning domain. We acknowledge that these additional tasks still rely of a similar structure, and discuss these limitations and directions for future work in Appendix A.

We address further feedback and questions point by point below.

Motivation & practicality: We thank the reviewer for these questions. In response, we add a set of evaluations in Appendix D (‘verbalization of beliefs’), which elicit the LMs’ beliefs about the user’s preferences through prompting (‘verbalization’). Similar analyses were conducted by Qiu et al. (2026), who show that BayesLM can better verbalize its beliefs than OracleLM or the untrained LMs. However, these analyses haven’t directly assessed the calibration of the verbalized beliefs. We compare beliefs accessed through verbalization to the model encoded information that we assess through probing. The verbalized beliefs show a considerably larger divergence than the probed beliefs, suggesting that verbalization does not fully retrieve the information the LM encodes. This suggests that Bayesian fine-tuning leads to more accurate Bayesian behavior and beliefs than standard fine-tuning, but still doesn't guarantee that verbalized beliefs are calibrated with respect to the models' encoded information. Full details are now reported in Appendix D.

Experimental limitations: We thank the reviewer for pointing this out, and now completed the statistical analyses and added remaining results for all backbones in the Appendices B, F, H.

Answers to the questions:

  1. We added these analyses in Tables 12–13. Indeed, BayesLM's accuracy wasn't significantly different from OracleLM (p>0.05), but StartLM was significantly worse (p<10-4). The performance across all test sets and models was not significantly different from the original test set (p>0.05).
  2. We added these analyses in Table 14–16, and now describe them in detail in Appendix H.
  3. We now include the full analysis in Figures 25–26.
  4. Qwen seems to perform worse throughout our analyses, including the strength of the encoded beliefs. We speculate that this might also be the reason for worse intervention results, as also reported in Qiu et al. (2026). We now updated Figures 16–17 to include a bootstrapped confidence interval (gray shaded area).
  5. The additional results are shown in Figure 28, and the statistical analyses are now reported in Tables 17–19.
AuthorsOpenReview ↗

Follow-up to Reviewer Miou

Dear Reviewer Miou,

As the author response period ends soon, we would be grateful for your feedback on our rebuttal. Please let us know if any concerns remain that we can still address.

Also, while responding to the other reviewers, we revised the manuscript, and this changed some table numbers. We have updated our response to you so its table references match the current version.

Thank you again for your time and constructive comments.

Reviewer_RtFdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether Bayesian fine-tuning helps language models make Bayesian decisions rather than simply copy Bayesian behavior. The authors compare models trained on Bayesian predictions with those trained on ground truth answers using a flight recommendation task. Through probing and intervention method, they show that Bayesian fine-tuning helps models better represent and use beliefs when making decisions.

Critical strengths
  • This paper's presentation is clear.
  • The proposed method uses probing and causal intervention to interprete why Bayesian fine-tuning helps models better represent and use beliefs when making decisions, providing useful insights.
Critical weaknesses
  • The authors implement a simple and small-scale task to support all their findings, and the four test sets they use are merely variations from one single task, which extremely restricts the generalizability of the analysis results and the scope of contribution.
  • This paper mainly combines existing techniques to analyse the difference between Bayes LM and Oracle LM, which limits its methodological contribution.
  • The paper mainly compares Bayes LM and Oracle LM. It is unclear whether the improvements come specifically from Bayesian supervision or simply from using more informative (or smoother) training signals.
Questions for authors
  • Could the authors conduct additional experiments on different tasks to better demonstrate the generalizability of the proposed framework?
  • Could the authors provide additional baselines, such as a model trained with soft labels, to better understand whether the improvements come specifically from Bayesian supervision?
3 comments
AuthorsOpenReview ↗

Framework clarifications and additional task results

We thank the reviewer for the helpful feedback.

To address weakness 1 / question 1, we emphasize that a key contribution of the paper is the framework for evaluating whether language models are Bayesian decision makers, rather than only the conclusion about whether a particular LM is Bayesian or how well it performs. Therefore, the task-specific results become less central, since we can argue that the same framework could be applied to other models even if they lead to different conclusions. Nonetheless, we now additionally report results on two additional tasks - a hotel recommendation task (Qiu et al., 2026), and a baseline nonsense task (added in Appendix I). While the hotel task tests the model’s generalization to a different domain, experiments on nonsense data allow to remove any semantic interference from possible in-weight world knowledge, and isolate the probabilistic reasoning task. It uses the same structure as the flight recommendation task, but uses nonsense feature names and simply calls the options “items”. Crucially, the key insights gained through our framework on the flight recommendation task are further strengthened by these additional analyses. The analyses on both tasks confirm that BayesLM outperforms the OracleLM behaviorally, although the gap decreased. The models also performed slightly worse on the nonsense task. Interestingly, probing results confirm that BayesLM learns to represent more accurate beliefs than OracleLM on both tasks, although the models haven’t been trained on these tasks. Second, the neuro-symbolic analysis shows a more substantial gap than on the flight recommendation task between both models’ performance and the neuro-symbolic accuracy, suggesting that the models may generalize well to representing probabilistic beliefs on new tasks (the neuro-symbolic accuracy comes close to BayesLM’s original flight accuracy), but that the reading of these beliefs for decision making might be sensitive to the fine-tuning domain.

Weakness 2: We agree that the individual techniques we use are well established, and we chose them deliberately, so as to have a reliable toolbox where interpretation and control of results are clear. Our main contribution is the framework: a complete, operational definition of what it means for an LM to behave as a Bayesian decision maker, and a way to test this in controlled tasks. The framework breaks this question down into concrete requirements. We argue that using established techniques in this setting is a methodological advantage. Because the tools are well understood and validated, any difference between the BayesLM and the OracleLM can be interpreted as differences in the level of satisfaction of the respective requirement by the model. It is unlikely to be explained by issues with unvalidated methods. A novel technique would add a confound that we avoid.

Weakness 3/ Question 2: The work by Qiu et al. (2026) from which the Bayesian fine-tuning procedure stems conducts extensive comparisons to baseline fine-tuning approaches, including varying the priors and fine-tuning with data that has the same proportion of errors, but included randomly, not generated by a Bayesian assistant. They show that Bayesian supervision, not mere presence of errors, leads to the advantage, and we would expect this specific advantage to percolate into the model’s beliefs and inference. While training models with soft labels is beyond the scope of the revisions, we include a discussion of this baseline as an important direction for future work (Appendix A).

Dear Reviewer RtFd,

As the author response period ends soon, we would be grateful for your feedback on our rebuttal. Please let us know if any concerns remain that we can still address.

Thank you again for your time and constructive comments.

Thank you for the clarification and additional results. They partially address my concerns about generalizability and contribution. However, my concern about the lack of alternative supervision baselines remains unresolved.

Therefore, I will maintain my score.

Reviewer_B9HDOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework to assess Bayesian behaviours of LLMs when true answers are missed. It includes four progressive requirements: act Bayesian (output follows Bayes's rule), ** represent Bayesian beliefs** (internal representations match predictions by Bayes's rules), ** act on Bayesian beliefs** (causal analysis between internal representations and final outputs), and ** do Bayesian computation** (policy probabilities matching Bayes optimality).

With a flight recommendation task, the paper tests four levels throughout: it tests acting Bayesian with behaviour analysis, where three models are compared on output accuracy: Bayesian fine-tuning, oracle fine-tuning, and vanilla baseline; it tests Bayesian representations by using probing to understand whether hidden states are Bayes-consistent; it then tests acting on Bayesian beliefs by ablation studies: what might be the impact of injected prior beliefs? Lastly, it tests whether/how final outputs of LLMs are actually informed by their beliefs by doing cross-model patching.

Through those experiments, they reached the conclusion that

(i) Bayesian supervision's advantage is that it improves the reasoning process, and such performance is superior to tuning with oracle-correct signals.

(ii) Behaviour is not enough to understand reasoning because it can not discriminate between wrong beliefs with wrong answers and correct beliefs leading to wrong answers because of erroneous reasoning.

Critical strengths

The presentation of this paper is among the best there can be. It presents a four-layer framework to test Bayesian belief updating in LLMs, and the paper also follows this progression while presenting substantial analysis of each layer. I enjoyed reading it throughout.

Critical weaknesses

Upon reading closely, the most salient weakness of this paper is it only tests Bayesian belief updating in one test domain (the flight recommendation task). As argued in more detail in "Questions For Authors", this may not be a domain where LLM beliefs may be distorted the most.

Questions for authors

Did you get a chance to conduct experiments in a different domain/test? This is an important question, especially when we consider that the conclusion from this paper is "its LLM does not encode Bayesian-optimal beliefs", rather than that encoded beliefs do not contribute to outputs according to Bayes's rules. There have been substantial studies showing that there are domain discrepancies when it comes to encoding Bayesian-optimal prior beliefs (e.g., https://arxiv.org/abs/2512.02914), likely because RLHF distorted LLM beliefs to increase user approval (https://arxiv.org/abs/2411.15287). Arguably, the flight recommendation task may not be the domain where LLM beliefs are distorted the most, in contrast to subjective domains such as "ChangeMyView".

2 comments
AuthorsOpenReview ↗

Additional task results

We thank the reviewer for the positive feedback. A key contribution of the paper is the framework for evaluating whether language models are Bayesian decision makers, rather than only the conclusion about whether a particular LM is Bayesian.Therefore, the specific result on whether the LM encodes Bayes-optimal beliefs becomes less central, since we can argue that the same framework could be applied to other models, as we demonstrate with results on additional backbones, even if they lead to different conclusions. Nonetheless, we now additionally report results on a second domain - a hotel recommendation task (see Appendix I). These additional analyses confirm that BayesLM outperforms the OracleLM behaviorally. Crucially, our framework allows us to gain several insights beyond the behavioral results. First, the probing results confirm that BayesLM learns to represent more accurate beliefs than OracleLM. Second, the neuro-symbolic analysis shows a more substantial gap than on the flight recommendation task between both models’ performance and the neuro-symbolic accuracy, suggesting that the models may generalize well to representing probabilistic beliefs on new tasks (the neuro-symbolic accuracy comes close to BayesLM’s original flight accuracy), but that the reading of these beliefs for decision making might be sensitive to the fine-tuning domain. We additionally include a discussion of the suggested papers in Appendix A.

Dear Reviewer B9HD,

As the author response period ends soon, we would be grateful for your feedback on our rebuttal. Please let us know if any concerns remain that we can still address.

Thank you again for your time and constructive comments.

Reviewer_pysyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Prior work showed that fine tuning a language model on the outputs of a Bayesian assistant gives near Bayesian behaviour on a flight recommendation task, while fine tuning on the true answers does not. This paper asks why. It states four requirements for a Bayesian decision maker and tests them on Gemma 2 9B tuned with a Bayesian signal, with oracle answers, and not at all. The Bayes trained model is more accurate against the Bayesian oracle. Its beliefs decode better from the middle layers. Patching a prototype belief into those layers changes its recommendation in the predicted direction on most instances, and moving beliefs between the two tuned models transfers part of the gap. A neuro symbolic plug in and two attention masks test how beliefs become a choice.

Where it sits (October 2026): decoding belief states from residual streams is now a crowded line. Belief states are linearly decodable across many models (arXiv 2609.17376), posteriors have been modelled as representation manifolds with their own interventions (arXiv 2602.02315), belief updating has been tracked across layers and turns (arXiv 2511.00617), BayesBench measures belief trajectories under evidence (arXiv 2606.30850), and a recent study reads posteriors from prompts (arXiv 2610.29331). The distinct part here is the layered requirements, the three model design with the training signal as the only difference, and the causal test on one task.

Critical strengths

The three model design is clean. The models share the backbone, the data and the prompts, so every difference in what decodes and what moves is about the training signal. The per position probing in Figure 12 is a more than welcome sanity check. At the feedback token the decoded policy is at chance for every model while the belief is already decodable, so the belief probe is not reading the coming choice.

The intervention is the paper's spotlight result. The edit moves the Bayes trained model's answer where Bayes' rule predicts and leaves it alone where it should, the same edit is near chance on the oracle model, and the prototype built from the untrained model moves the Bayes model to chance in Figure 6. The simulator and task files are released. I could rerun the assistant on all 624 held out conversations and recover the recommendations, the beliefs and the flip annotations.

Critical weaknesses

The noise consistency claim for beliefs is contradicted by the paper's own numbers. Section 4 says the decoded beliefs of both tuned models respond adequately to noisy input. Appendix D reports a belief divergence of 0.52 and 0.58 on the two noisy sets, then rescues it with an exponential scaling of the probed beliefs that gives 0.083 and 0.059. I computed the divergence between the assistant's belief and a uniform distribution from the released task files. It is 0.083 on the low noise set and 0.059 on the high noise set, identical at three decimals. The scaling flattens the probe output to uniform. The rescued numbers are what any constant predictor scores, the raw numbers are six to ten times worse than uniform, and the abstract and Table 2 still count R2 as met.

Recommendation: add the uniform reference per test set to Figure 11, drop the scaled numbers, and correct the sentence in Section 4 and the claim in Appendix D.

The neuro symbolic argument for R4 does not survive its own control. Table 1 reads the gain of the plug in over the model's own accuracy as unread belief, 0.02 for the Bayes model and 0.09 for the oracle model. Figure 20 shows the same plug in reaching about 0.55 for the untrained model, whose own accuracy is 0.35. By the paper's logic the untrained model holds the most unread Bayesian beliefs. What the plug in measures is how much of the assistant's belief a probe trained on that belief can pull from a state. Table 7 adds that the ordering reverses on Qwen 2.5 7B, and that each backbone is reported at its best layer while Appendix G says the main numbers use the last layer.

Recommendation: add the untrained model and a label shuffled probe to Table 1, fix one layer for all models, and reword R4 as a bound.

The intervention numbers are conditioned and the matched controls are missing. The analysis code states that every rate is computed only on conversations whose unsteered answer already matched the oracle, and Table 6 confirms it. The caption says 100 flip and 100 stay instances, but every rate in the table is a multiple of 1/82 or 1/85. Appendix E says all 624 instances are used. So the 0.77 of the Bayes model is scored on the 84 percent where it was already right, the 0.52 of the oracle model on its own 69 percent. The conditioning is defensible, but the subsets differ per model and the paper does not say so. The same code computes two controls the text never reports, the mean state of the opposite belief band and the global mean state at the same position, both at the same dose. Its own documentation calls the second one the control that separates writing a belief from overwriting the state with anything. The paper reports only a norm matched random vector, which is off manifold and predictably does nothing. The masking result has a similar problem. In the code the sequential mask lets the final suggestion see one feedback span while the one swoop mask lets it see four. Under the sequential mask the Bayes model stays near 0.57 from round one to round four in Figure 18, the accuracy of a model that has seen a single data point. The 28 point drop measures how much evidence the mask removes.

Recommendation: state the conditioning with the number of instances per cell and add the unconditioned balanced accuracies, report the two prototype controls in Table 8, and add a mask that keeps the sequential structure but exposes one extra earlier feedback, or rewrite the claim as the model needing direct attention to every feedback span.

Questions for authors

Questions that could change my assessment:

What are the balanced accuracies of the opposite band and global mean prototype arms at layers 18 to 25? If the global mean moves the Bayes model substantially, R3 is a disturbance effect.

What is the neuro symbolic accuracy of the untrained model and of label shuffled probes at the layer used for Table 1, and does the Bayes versus oracle ordering hold on all three backbones at one fixed layer?

What are the unconditioned balanced accuracies, the number of instances per cell and model, and the exact denominators behind Table 6?

What is the decoded belief divergence on the noisy sets without the scaling, next to the uniform reference?

Recommendations (not affecting my score):

Release the result files. The repository ignores results, activations and the probing cache, so every table and figure script reads a file that is not there (?).

Use one averaging convention. Section 6 and Figure 6 average four settings and report 0.52 to 0.61 and 0.77 to 0.61, while Table 8 averages eight and gives 0.536 and 0.612, without the oracle to Bayes direction.

Fix p = 0.022 in Section 3 against 0.002 in Table 3, 0.68 in the text against 69 percent in Figure 19, and the two identical rows in Table 6.

Cluster the standard errors of the Table 3 regression by conversation. The four test sets are edits of the same 624 conversations and the three models answer the same prompts.

Report the control task numbers Appendix A refers to. Every released launcher passes control task NONE.

Choose beta and the layer window on a slice of the training conversations. Beta 4 and the window 18 to 25 are both the optimum on the evaluated set.

4 comments
AuthorsOpenReview ↗

Additional results & Clarifications

We thank the reviewer for the detailed feedback, actionable suggestions and questions. We address the recommendations and questions one by one below.

Recommendation 1 / Question 4: We thank the reviewer for pointing out the result with the uniform control. We improve the discussion in Appendix D when describing the generalization of the probes trained on the original dataset. We further clarify that the high divergence of these probes and the increase in their divergence in after layer 15 shown in Fig.11 are likely artifacts of the probe training, rather than the model’s fundamental failure to represent noisy beliefs. To confirm this, we train belief probes on a noisy training set for BayesLM, and include the new Figure 14 in Appendix D. These results are comparable to the divergences of the original probes for the original dataset. We normalize the results by the uniform distribution baseline in Figure 14 in response to the reviewer’s suggestion to include the uniform baseline. We clarify the sentence in Section 4 that the probes trained on different training data perform adequately on the respective test sets.

Recommendation 2: We now include the results from the base model that were only reported in Fig 20 in Table 1. We correct the typo in Appendix G, as all results were reported for the respective best layer of each model. We now additionally report neuro-symbolic results for all models across all layers in Figure 22, which shows that the neuro-symbolic accuracies do not largely differ across layers, after approximately layer 15, making the particular layer choice secondary. The BayesLM outperforms OracleLM across layers for both Gemma and Llama, but not Qwen, which performed worse throughout the evaluations, as also shown by Qiu et al. (2026). We argue that using the trained probes in the neuro-symbolic set-up of R4 goes beyond assessing how much a trained probe can pull from a state: this is assessed in R2 by directly comparing the probes to the assistant’s beliefs. R4 additionally addresses how well the task can be performed under the ground truth decision rule from the pulled beliefs, and the comparison to the performance of the model which represents that information arguably isolates the approximation of the decision rule by the model. Although this hinges on the quality of the probes, evaluations and selectivity controls reported for R2 suggested reliable probes, providing an upper bound on the Bayesian decision making performance that we might expect from a model with the given beliefs.

Recommendation 3 / Question 3: We would like to clarify that across main reported patching results, all models and backbones were tested on the full original test set of 624 samples, without any subsetting. We further clarify that 100 samples per flipping condition were only used for the hyperparameter search, reported in Appendix E, and these were 100 separately sampled items. Therefore, the denominator of each cell in Table 6 is 200. We clarify this in the text. Addressing Question 1, we now report the requested additional controls in Appendix E (‘additional controls’ paragraph). The lower accuracy under the global mean suggests that the intervention is not a mere disturbance effect, but indeed results from the intended belief intervention. We additionally expand the description of the sequential vs. one-swoop mask, adding that the bigger drop in performance under the sequential mask is also consistent with the interpretation that the model needs direct attention to every feedback span (lines 347–349).

Responses to recommendations not affecting the score (addressing them on by one):

  1. We will release instance-level results files. Full probing caches are too large for permanent public storage.
  2. Fixed.
  3. Fixed.
  4. We kindly ask the reviewer to clarify what is meant by ‘cluster the standard error’.
  5. We now explain the controls in Section 4 when introducing the probing set up.
  6. We thank the reviewer for this suggestion, and will include it in future work.
Reviewer_pysyOpenReview ↗

Response to author's rebuttal

Thank you for the additional controls and the noisy-probe experiment. The global-mean prototype staying at chance while the opposite-direction prototype reverses the recommendation supports R3 as a belief effect rather than a disturbance. The retrained noisy probes also address my concern about Appendix D.

My assessment is unchanged, for the reasons below.

Denominators in Table 6. The response states 100 instances per condition. Every flip rate in the revised table is a multiple of 1/82 and every stay rate a multiple of 1/85 (for example 36/82 and 81/85), including the noise column. These counts match the share of instances the unsteered BayesLM already answers correctly, which is the conditioning I described from the code. The response therefore does not match the paper. Please state which instances enter each rate, with the count per model and cell, for Table 6 and for the 624-instance results in Table 9.

Clustered standard errors. The rows of the Table 3 regression are not independent. The four test sets are edits of the same 624 conversations and the three models answer the same prompts, so each conversation contributes up to twelve correlated observations. Treating them as independent makes the standard errors too small. A mixed-effects logistic regression with a random intercept per conversation, or cluster-robust standard errors with the conversation as the cluster, would fix this.

Consistency of the revised text.

  • Section 4 says StartLM shows none of the belief signal, while Table 1 now gives it the largest neuro-symbolic gain. This bears directly on reading the OracleLM gain as unread belief.
AuthorsOpenReview ↗

Denominator information, patching analysis on full test set, statistical results with random effects and text edits

We thank the reviewer for taking into account our response and the detailed follow-up. We provide responses below.

Denominators in Table 6. We thank the reviewer for pointing out the inconsistency, and acknowledge that our report was incorrect. The detailed feedback helped us catch the error, adjust the explanations in the paper and motivated additional analyses. Indeed, the reported results for the intervention apply the intervention only on instances where the models were originally correct. These instances are drawn from the full original test set for the main results (Fig. 6, Fig. 16-17, formerly Tab. 7, now Tab. 11), and from separate 100 instances per flipping condition that we sampled for hyperparameter tuning (Tab. 6). We now report the exact number of instances in each cell (the denominators) in the table to Appendix E, a new Table 8.

The previous response mistakenly indicated that we did not subset the sample, so we apologize for the error. The error was because as part of the experiments, we explored both approaches: evaluating the patching on subsets and on the full test data. We keep the main results on instances where the models were correct, and now explicitly state this in Section 5 (lines 348-350). As also confirmed by the reviewer, we think this is, after all, a defensible design choice for identifying the role of the beliefs in recommendations, without potential error interference of incorrect prediction instances that may be due to other factors than beliefs. This design decision does not affect the key take-aways. We now also report patching results on the full test set of 624 instances, without subsetting, shown side-by-side with the main results in the table below (included in Appendix E, Table 9).

Donor → recipientMethodBal. acc. ABal. acc. B
BayesLM→BayesLMpatching0.77 [0.76, 0.78]0.76 [0.75, 0.77]
OracleLM→OracleLMpatching0.53 [0.52, 0.55]0.51 [0.50, 0.53]
BayesLM→OracleLMpatching0.61 [0.60, 0.63]0.59* [0.58, 0.60]
OracleLM→BayesLMpatching0.62 [0.60, 0.63]0.62 [0.60, 0.63]
BayesLM→BayesLMcontrol: random (norm-matched)0.50 [0.49, 0.50]0.49 [0.48, 0.50]
BayesLM→BayesLMcontrol: opposite-direction prototype0.22 [0.21, 0.23]0.21 [0.19, 0.22]
BayesLM→BayesLMcontrol: global-mean prototype0.48 [0.47, 0.49]0.48 [0.47, 0.49]
BayesLM→BayesLMsteering (CAA)0.59 [0.58, 0.60]0.59 [0.58, 0.60]

Table 9: Comparing the balanced accuracy of belief patching on a subset (only original test set instances where models' unpatched matched BayesAssist, reported in Section 5, to balanced accuracy on all instances. ∗^* indicates when the two results are significantly different. Intervals are bootstrapped over conversations (10{,}000 resamples).

The results and conclusions remain qualitatively the same, and quantitatively the results remain close. Under this evaluation, BayesLM still outperforms OracleLM, and patching BayesLM’s beliefs into OracleLM still leads to a gain in balanced accuracy compared to OracleLM’s own patching. The patching still outperforms CAA and the controls.

Clustered standard errors. We thank the reviewer for the clarification. We now update the regression model in Table 3 to include random by-conversation intercepts (regression formula: correct ~ C(model) * C(set) + (1 | conversation)). The results remain qualitatively the same. We additionally updated all the other reported regression analyses to also include the random intercept, and also observe no qualitative changes.

Consistency of the revised text. We thank the reviewer for pointing out the mismatch in the text. We assume that the last sentence refers to the StartLM (not OracleLM) gain. We update the text in lines 282-285 discussing results in Fig. 4.

Dear Reviewer pysy,

As the author response period ends soon, we would be grateful for your feedback on our additional response.

Thank you again for your time!

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.