- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper trains a two-layer transformer network to infer a common cause from data sampled for a causal Bayesian network. It compares the transformer's behavior to an idealized Bayesian learner, showing that the transformer behaves like an ideal Bayesian learner on in-distribution data, as well as on edits to the input variables and, to a lesser extent, generalization to sequences with a latent common cause ("held-out" sequences). It then provides a mechanistic analysis of the transformer, showing that its embeddings encode the identity and values of different variables, and that the residual stream encodes the direction of evidence about the common cause. The authors show that the residual stream causally represents this Bayesian belief about the common cause through interventions on the residual stream. The mechanistic analysis explains the model's generalization to the held-out context, and additionally explains why it fails to generalize as well as an ideal Bayesian learner.
Overall, this paper shows how a small transformer could implement dynamic Bayesian inference about a latent common cause. Although it remains an open question as to how/whether larger transformers implement Bayesian inference, this result demonstrates that it is at least possible for large models to do so.
Critical strengths
- Tracking common causes is relevant for the formation of world models. Understanding how models could represent Bayesian evidence for latent common causes can help us understand what information they might infer from context
- The toy task is relevant for analyzing how models learn to infer latent variables, and allows the authors to control the evidence the model receives to predict the common cause, while controlling for confounds such as token position
- The paper employs thorough mechanistic analysis that uses correlational evidence, causal interventions, and reasoning about the model's architecture to understand how it implements Bayesian inference in a toy setting
Critical weaknesses
- The writing is difficult to parse (see below).
- Many terms aren't clearly defined, such as: "value plane", "variable plane", "principle angle analysis", "belief unit", "exchangeability of variables and parents screen off children" (page 4), "Bayesian-like" (used throughout paper), "prediction deficit", "gain"/"shrinkage"/"accumulation gain"
- The figures are hard to read. For instance, Figure 6 includes two figures with a double y-axis. It isn't clear what the labels for the additional axes are. For the remaining axes, all labels use mathematical expressions that aren't clearly explained in the main text.
- The paper focuses on a single toy task and a two-layer transformer trained on it. Testing on a larger model or on an extension of the toy task would be helpful in understanding what to take away from the experiments.
- Although the research questions are explicitly stated, it's unclear what alternative hypotheses the authors consider.
Questions for authors
- I'm not quite sure I understand what is meant by "held-out contexts" and "held out sequences". Page 4 says, "In the held-out context, the parent is removed from the loss and from the attention keys whenever it follows both children". This is confusing, since "held-out" tends to mean "held out from training", but this isn't what's happening here. What exactly is "held out" for these data points?
- In addition to inferring the common cause, it seems that the model must infer (or memorize) the characters, since these change between contexts. Could a hierarchical Bayesian model account for this, and potentially explain the re-use of token embeddings for generalizing to held-out contexts?
3 comments
I appreciate that the authors took the time to rewrite main sections of the paper to improve clarity. The updated Figure 1 and terminology definitions are particularly helpful.
That said, the authors did not update the figures as they stated in the response ("The right-hand parts of Figures 6b and 6c are separate panels" - Figure 6 did not change as far as I can tell). The alternative hypotheses are also not quite clear to me. For instance, what would it look like for a "belief coordinate" to not "emerge after attention"?
For these reasons, I maintain my decision of weak rejection.
Response to the Comment by Reviewer jzt6 (follow-up)
General. We thank the reviewer for reading the revision closely and for the two remaining points, which we address below.
"Figure 6 did not change as far as I can tell."
You are right, and we apologize. Our response was inaccurate on this point. The first revision only changed the caption of Figure 6, not the figure. We have now revised the figure itself and rewritten its caption to match it and to assist the reading of each panel. In the uploaded version:
- The two double y-axes are gone. The former right-hand parts of panels (b) and (c) are now the separate panels (d) and (e), each with its own axis.
- The axis labels are in words. The only symbols left on them are ones that we explicitly define in the main text.
- The caption now states what each panel shows and explains the axis labels.
The diff PDF is updated accordingly.
"What would it look like for a belief coordinate to not emerge after attention?"
Section 5.2 considers two ways in which a belief coordinate could have failed to emerge, and rules out both.
- The head compares the two tokens (opening paragraph). Ruled out because the attention pattern does not depend on the input (Appendix D).
- The head applies a rotation, so that the values of the two tokens end up along different directions (alignment paragraph). Ruled out because their value axes are almost perfectly aligned after the head (Appendix E).
What remains is an attention block that approximately averages the two last tokens of a sequence along a shared value direction (Eq. 2 and the text that follows it). The three values of the belief coordinate follow from this, as the last paragraph of Section 5.2 and Figure 4b show: two children with the same value average to the position of one child, and two children with different values average to zero. Section 5.3 then shows where the amount of evidence is supplied, and Section 6 tests the belief coordinate by intervention.
Response to Reviewer jzt6
General. We thank the reviewer for the careful reading and the accurate summary. All points concern the presentation, and we have revised the paper accordingly. Below we point to each change. We have also uploaded a diff PDF that highlights all changes relative to the original submission, so that the revisions to the parts of interest can be traced directly.
The paper is thoroughly revised, in particular Sections 3 to 6. The masking procedure and a formal definition of Bayesian generalization are now in Section 3, and the new top row of Figure 1 shows the experimental setup.
Each term is now defined in the sentence that introduces it:
We replaced "principal angle" with "alignment" in the main text, as the technical jargon is unnecessary to understand the result, and define the principal angles themselves in Appendix E. We also dropped "Bayesian-like" and now state the two claims it stood for explicitly, in Section 3 (Bayesian generalization) and Section 6 (the represented latent operates as a Bayesian latent).
Figure 1 has a new top row that shows the dependency structure, the training sequences and the held-out position. The right-hand parts of Figures 6b and 6c are separate panels, and the revised caption of Figure 6 names each panel and each axis in words. The caption of Figure 2 now defines the baselines.
We agree that this limits what can be concluded, and we do not claim that larger models use this mechanism. What the small scale makes possible is a complete account of how Bayesian generalization comes about, traced from the embeddings to the output and tested by intervention, which gives a concrete hypothesis about the representations and the mechanism to look for in larger models (Section 7). Within the paper, the result holds across 45 models, three held-out contexts, and both layers in which the functional attention can emerge (Appendix A). Extensions of the task are now named as future work in Section 7.
Each alternative is now stated where it is tested. Here is a very compressed summary of the alternative hypotheses:
The term invited the wrong reading and we have changed it. Section 3 now defines the held-out context, the held-out positions and the held-out posterior, and states what remains supervised. Figure 1 (top row) shows the one position that is held out next to the sequences that are trained.
This is a good point. The reviewer is right that the model first has to work out which variable and which value each token stands for. We had left this implicit and now state it in Section 3. Since each token stands for the same variable and value in every sequence, there is nothing to infer about this within a sequence. The model learns the assignment in its embeddings, and Section 5.1 shows that the embeddings of every context, including the held-out one, are organized in the same way, which is why the same mechanism applies to the held-out context. A hierarchical Bayesian model would be needed if the mapping changed between sequences, so that the model had to infer it on the fly. We find this an interesting extension and mention it as future work in Section 7.