Transformers track Bayesian evidence for latent common causes via a context-invariant mechanism

We present an in-depth investigation of how a form of Bayesian causal reasoning about common causes can emerge as a cross-contextual generalization in small, but fully tractable transformers. Incrementing on like-minded recent work, our set-up (i) disentangles causal mechanisms in the model from the causal structure of the true data-generating processs, (ii) orients more towards natural language prediction by considering inference of latent common causes, and (iii) considers whether and how Bayesian evidence accumulation for latent common causes can be implemented in representations and mechanism that allow for cross-context generalization to novel test cases.