acceptodds
Under review as a conference paper at ICLR 2027

Transformers track Bayesian evidence for latent common causes via a context-invariant mechanism

Abstract

We present an in-depth investigation of how a form of Bayesian causal reasoning about common causes can emerge as a cross-contextual generalization in small, but fully tractable transformers. Incrementing on like-minded recent work, our set-up (i) disentangles causal mechanisms in the model from the causal structure of the true data-generating processs, (ii) orients more towards natural language prediction by considering inference of latent common causes, and (iii) considers whether and how Bayesian evidence accumulation for latent common causes can be implemented in representations and mechanism that allow for cross-context generalization to novel test cases.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, Bayesian inference, transformers, generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jzt6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper trains a two-layer transformer network to infer a common cause from data sampled for a causal Bayesian network. It compares the transformer's behavior to an idealized Bayesian learner, showing that the transformer behaves like an ideal Bayesian learner on in-distribution data, as well as on edits to the input variables and, to a lesser extent, generalization to sequences with a latent common cause ("held-out" sequences). It then provides a mechanistic analysis of the transformer, showing that its embeddings encode the identity and values of different variables, and that the residual stream encodes the direction of evidence about the common cause. The authors show that the residual stream causally represents this Bayesian belief about the common cause through interventions on the residual stream. The mechanistic analysis explains the model's generalization to the held-out context, and additionally explains why it fails to generalize as well as an ideal Bayesian learner.

Overall, this paper shows how a small transformer could implement dynamic Bayesian inference about a latent common cause. Although it remains an open question as to how/whether larger transformers implement Bayesian inference, this result demonstrates that it is at least possible for large models to do so.

Critical strengths
  • Tracking common causes is relevant for the formation of world models. Understanding how models could represent Bayesian evidence for latent common causes can help us understand what information they might infer from context
  • The toy task is relevant for analyzing how models learn to infer latent variables, and allows the authors to control the evidence the model receives to predict the common cause, while controlling for confounds such as token position
  • The paper employs thorough mechanistic analysis that uses correlational evidence, causal interventions, and reasoning about the model's architecture to understand how it implements Bayesian inference in a toy setting
Critical weaknesses
  • The writing is difficult to parse (see below).
  • Many terms aren't clearly defined, such as: "value plane", "variable plane", "principle angle analysis", "belief unit", "exchangeability of variables and parents screen off children" (page 4), "Bayesian-like" (used throughout paper), "prediction deficit", "gain"/"shrinkage"/"accumulation gain"
  • The figures are hard to read. For instance, Figure 6 includes two figures with a double y-axis. It isn't clear what the labels for the additional axes are. For the remaining axes, all labels use mathematical expressions that aren't clearly explained in the main text.
  • The paper focuses on a single toy task and a two-layer transformer trained on it. Testing on a larger model or on an extension of the toy task would be helpful in understanding what to take away from the experiments.
  • Although the research questions are explicitly stated, it's unclear what alternative hypotheses the authors consider.
Questions for authors
  • I'm not quite sure I understand what is meant by "held-out contexts" and "held out sequences". Page 4 says, "In the held-out context, the parent is removed from the loss and from the attention keys whenever it follows both children". This is confusing, since "held-out" tends to mean "held out from training", but this isn't what's happening here. What exactly is "held out" for these data points?
  • In addition to inferring the common cause, it seems that the model must infer (or memorize) the characters, since these change between contexts. Could a hierarchical Bayesian model account for this, and potentially explain the re-use of token embeddings for generalizing to held-out contexts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jzt6

General. We thank the reviewer for the careful reading and the accurate summary. All points concern the presentation, and we have revised the paper accordingly. Below we point to each change. We have also uploaded a diff PDF that highlights all changes relative to the original submission, so that the revisions to the parts of interest can be traced directly.

"The writing is difficult to parse."

The paper is thoroughly revised, in particular Sections 3 to 6. The masking procedure and a formal definition of Bayesian generalization are now in Section 3, and the new top row of Figure 1 shows the experimental setup.

"Many terms aren't clearly defined."

Each term is now defined in the sentence that introduces it:

  • exchangeability, screening off and belief unit in Section 3 (canonical states),
  • variable plane and value plane in Section 5.1,
  • belief coordinate in Section 5.2,
  • prediction deficit, shrinkage and accumulation gain in Section 5.3,
  • unit belief in Section 6.

We replaced "principal angle" with "alignment" in the main text, as the technical jargon is unnecessary to understand the result, and define the principal angles themselves in Appendix E. We also dropped "Bayesian-like" and now state the two claims it stood for explicitly, in Section 3 (Bayesian generalization) and Section 6 (the represented latent operates as a Bayesian latent).

"The figures are hard to read."

Figure 1 has a new top row that shows the dependency structure, the training sequences and the held-out position. The right-hand parts of Figures 6b and 6c are separate panels, and the revised caption of Figure 6 names each panel and each axis in words. The caption of Figure 2 now defines the baselines.

"A single toy task and a two-layer transformer."

We agree that this limits what can be concluded, and we do not claim that larger models use this mechanism. What the small scale makes possible is a complete account of how Bayesian generalization comes about, traced from the embeddings to the output and tested by intervention, which gives a concrete hypothesis about the representations and the mechanism to look for in larger models (Section 7). Within the paper, the result holds across 45 models, three held-out contexts, and both layers in which the functional attention can emerge (Appendix A). Extensions of the task are now named as future work in Section 7.

"Unclear what alternative hypotheses the authors consider."

Each alternative is now stated where it is tested. Here is a very compressed summary of the alternative hypotheses:

  • Section 4: the variable of a token is read from its position.
  • Section 5.2: the attention head compares the tokens; the head applies a rotation.
  • Section 5.3: the belief coordinate carries the amount of evidence; the amplification comes from other components; the shrinkage lies outside the variable plane.
  • Section 6: a computation with no latent shared between parent and sibling.

"What exactly is 'held out'?"

The term invited the wrong reading and we have changed it. Section 3 now defines the held-out context, the held-out positions and the held-out posterior, and states what remains supervised. Figure 1 (top row) shows the one position that is held out next to the sequences that are trained.

"Could a hierarchical Bayesian model account for this?"

This is a good point. The reviewer is right that the model first has to work out which variable and which value each token stands for. We had left this implicit and now state it in Section 3. Since each token stands for the same variable and value in every sequence, there is nothing to infer about this within a sequence. The model learns the assignment in its embeddings, and Section 5.1 shows that the embeddings of every context, including the held-out one, are organized in the same way, which is why the same mechanism applies to the held-out context. A hierarchical Bayesian model would be needed if the mapping changed between sequences, so that the model had to infer it on the fly. We find this an interesting extension and mention it as future work in Section 7.

I appreciate that the authors took the time to rewrite main sections of the paper to improve clarity. The updated Figure 1 and terminology definitions are particularly helpful.

That said, the authors did not update the figures as they stated in the response ("The right-hand parts of Figures 6b and 6c are separate panels" - Figure 6 did not change as far as I can tell). The alternative hypotheses are also not quite clear to me. For instance, what would it look like for a "belief coordinate" to not "emerge after attention"?

For these reasons, I maintain my decision of weak rejection.

AuthorsOpenReview ↗

Response to the Comment by Reviewer jzt6 (follow-up)

General. We thank the reviewer for reading the revision closely and for the two remaining points, which we address below.

"Figure 6 did not change as far as I can tell."

You are right, and we apologize. Our response was inaccurate on this point. The first revision only changed the caption of Figure 6, not the figure. We have now revised the figure itself and rewritten its caption to match it and to assist the reading of each panel. In the uploaded version:

  • The two double y-axes are gone. The former right-hand parts of panels (b) and (c) are now the separate panels (d) and (e), each with its own axis.
  • The axis labels are in words. The only symbols left on them are ones that we explicitly define in the main text.
  • The caption now states what each panel shows and explains the axis labels.

The diff PDF is updated accordingly.

"What would it look like for a belief coordinate to not emerge after attention?"

Section 5.2 considers two ways in which a belief coordinate could have failed to emerge, and rules out both.

  1. The head compares the two tokens (opening paragraph). Ruled out because the attention pattern does not depend on the input (Appendix D).
  2. The head applies a rotation, so that the values of the two tokens end up along different directions (alignment paragraph). Ruled out because their value axes are almost perfectly aligned after the head (Appendix E).

What remains is an attention block that approximately averages the two last tokens of a sequence along a shared value direction (Eq. 2 and the text that follows it). The three values of the belief coordinate follow from this, as the last paragraph of Section 5.2 and Figure 4b show: two children with the same value average to the position of one child, and two children with different values average to zero. Section 5.3 then shows where the amount of evidence is supplied, and Section 6 tests the belief coordinate by intervention.

Reviewer_3YNDOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

"This paper seeks to contribute to the investigation of emergence of world models by an in-depth, mechanistic analysis of generalizable Bayesian (causal) reasoning emerging in a novel, minimalistic experimental setup. "

Main contribution: (i) a set-up (ii) data-generating process with common causes, (iii) “Bayesian generalization” for inferring a latent common cause in novel test cases.

Critical strengths

None

Critical weaknesses

The use-case is unclear. When and how can I use this new setup? What are the advantages? What would be my motivation?

Was there any additional stronger baselines to compare against? I see some comparisons with uniform and random - is this the best baseline?

Questions for authors

What is your formal definition of "Bayesian generalization" ?

Are there any guarantees that the setup and claims would hold in other settings?

  • questions from before The use-case is unclear. When and how can I use this new setup? What are the advantages? What would be my motivation?

Was there any additional stronger baselines to compare against? I see some comparisons with uniform and random - is this the best baseline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3YND

General. We thank the reviewer for the review. Since no specific soundness issue has been raised, we directly answer the points raised. We have also uploaded a diff PDF that highlights all changes relative to the original submission, so that the revisions to the parts of interest can be traced directly.

"The use-case is unclear. When and how can I use this new setup? What are the advantages? What would be my motivation?"

We would like to stress that our paper targets foundational research and understanding, as many ICLR submissions do. It stands in line with an actively investigated area, part of which is represented at ICLR (e.g., Li et al., 2023; Xie et al., 2022), and makes a novel contribution to it. Our setup is a minimal instance of the latent-variable problem posed in Section 2, which is quite similar to that of natural language. We directly extend prior work, such as on belief geometry (e.g., Shai et al., 2024; Piotrowski et al., 2025) and on learned causal structure (e.g., Nichani et al., 2024), by investigating the smallest data-generating process with a latent common cause, such that a learner is forced to carry a belief about this common cause to be able to generalize in the held-out context. As far as we can tell, and as mentioned in the paper, the detailed study of the mechanics of generalization as a Bayesian process is a novelty and provides the basis for predictions about similar mechanisms in larger models for follow-up work (Section 7).

"Was there any additional stronger baselines to compare against? I see some comparisons with uniform and random - is this the best baseline?"

Yes, there were, and the new completely reworked version of the paper makes that more transparent thanks to this suggestion. We report a KL-divergence to the Bayesian ground-truth itself, therefore the reference point is the optimal predictor at KL=0. The two baselines are predictors that cannot accumulate Bayesian evidence: one is uniform over the tokens of the held-out context, the other is uniform over its two parent tokens. Our behavioral analysis aims to measure the distance of the prediction success in the held-out context to the Bayesian ground-truth, with an eye on how far it sits from the other end of the spectrum. We are not aware of any competing models that use this exact setup. Section 4 and the caption of Figure 2 now state what each reference point measures.

"What is your formal definition of 'Bayesian generalization'?"

Thank you for pointing this out. Section 3 now gives the formal definition, in the paragraph "Bayesian generalization", and Figure 1 (top row) shows the posterior it concerns.

"Are there any guarantees that the setup and claims would hold in other settings?"

We do not claim a novel formal guarantee. We cite sources showing that an optimal predictor of a generative process is Bayes-optimal (Section 3), which justifies our modeling choice. As argued above, a small case study of this kind provides the scaffolding for follow-up research in larger settings: it yields concrete hypotheses about the representations and the mechanism to look for there. Section 7 now states this hypothesis for larger models and names other graph structures as future work.

Thanks the authors for updating the text and answering the questions. I'll update my score to 2 - weak reject.

Reviewer_pwNxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether a transformer model can actually learn Bayesian reasoning and what is the underlying mechanism for a transformer to learn such reasoning behavior. This paper starts from a simple fork structure in causal graph and test the reasoning ability of trained transformer by KL divergence on held-out positions. This paper first find out that trained model can learn the roles (two equivalence children and parent) of the fork structure. By visualizing the embeddings, this paper finds that embeddings space can learn the variable identity as well as variable value. Furthermore, this paper find out the roles of attention layer (approximate average of the two variables in the fork) and LayerNorm (normalization accounts for the mechanism of how more evidence contributing to stronger prediction), which explains the mechanism of Bayesian reasoning.

Critical strengths
  1. The design of experiment is very intuitive and the writing is clear. This paper begins with a very simple setting of fork structure and explores many aspects of such structure: equivalence of two children (Y1, Y2), parents screen off children, and how the variable relation strength plays the role. This paper further uses visualization to illustrate the variable role, which is intuitive.

  2. This paper explains the mechanism of attention layer and Layer- Norm in an intuitive way. Interestingly, this paper shows that attention layer aggregate the signals of tokens in a way similar as taking average (intuitively make sense after we take log-odds transformation) and the Layer-Norm helps to “count the evidence” in Bayes inference.

  3. This paper also uses intervention to cross check their findings about attention layer and Layer-Norm, which tests if one sibling’s residual activation changes, how the parent and the other sibling prediction change. I think this provides another rigorous proof to the previous interesting findings.

Critical weaknesses
  1. Although mechanism in Attention Layer and Layer-Norm is interesting, we do not know whether it would be the same on other classic causal graph structures. For instance, regarding the findings in Section 5.2 and Section 5.3, will the similar roles stay in a “collision” (X1, X2 independently causes Y) and “fully connected” (X has causal edge to Y1 and Y2; Y1 also has an causal edge to Y2). I understand the setting may require more complex data generation process, such as the variables no longer taking binary values, but it is worth exploring to prove the value of this paper.
Questions for authors

See Weakness Point 1.

1 comment
AuthorsOpenReview ↗

Response to Reviewer pwNx

General. We thank the reviewer for the accurate summary and the positive assessment, and address the one issue raised. We have also uploaded a diff PDF that highlights all changes relative to the original submission, so that the revisions to the parts of interest can be traced directly.

"…we do not know whether it would be the same on other classic causal graph structures…"

We agree that the generality of the mechanism across graph structures is the natural next question. We now name the two structures the reviewer mentions, as well as non-binary variables, as future work in Section 7.

We chose the fork for a number of reasons, most notably because:

  1. common causes are, as argued in Section 2, likely operative in natural language (e.g., a plural feature causing plural marking in a determiner and a noun), and
  2. it is the simplest graph in which the evidence from the observed variables adds up, which is what makes the held-out posterior follow from the supervised ones (Section 3).

A collider and a fully connected graph each remove one of the properties the mechanism relies on, the additivity of evidence in the first case and the exchangeability of the children in the second, so both are informative tests of it. Each needs its own data-generating process and analysis, which is why we leave them to a follow-up study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.