acceptodds
Under review as a conference paper at ICLR 2027

LEARNING SOURCE STRUCTURE: BAYESIAN PREDICTION WITH TRANSFORMERS

Abstract

A transformer trained on sequences whose source parameters are drawn afresh each time can learn only what the sequences share: the source family and its prior. The Bayesian predictor that knows both, which we call informed Bayes, is the unique minimizer of the training loss and is computable from the family alone; we use it as a quantitative, falsifiable reference. Where earlier work compared transformers with Bayes on sources whose contexts can be learned independently, we build Markov sources with up to 64 symbols in which observations in one context inform prediction in another: product sources with known or hidden encodings, transition rows sharing an unknown base law and depth, and a mixture of three families. Informed Bayes is evaluated numerically from the row integrals of the layered simplex architecture. Our main result is that informed Bayes predicts, from the family and prior alone, how a trained transformer behaves in four tests that go beyond the learning curve: the loss that remains under a source outside the training family (predicted 2.237 bits, measured 2.309), the doubling of the extra loss when two independent components change rather than one (0.345 predicted, 0.350 measured), the benefit of other contexts when a token appears for the first time, and the depth at which copying an earlier continuation starts to pay. On the learning curves themselves, small transformers match informed Bayes when the shared structure is visible in the tokens, and hidden structure needs more layers and training, with gaps below 0.01 bits per token for the largest models. We explain why the imitation is possible and where it must fail: informed Bayes decomposes into sufficient statistics that attention selects and accumulates and an estimator, fixed by the prior, that the feed-forward network learns; in the architecture-as-prior framework, training pays the complexity of informed Bayes under the architecture’s prior, and finite-memory universal prediction bounds how closely a bounded network can reach it.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
In-context learning, Transformers, Bayesian, Universal Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q9hPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a study comparing transformer model predictions with those of an informed Bayes model, that is, a model that knows both the source family and the prior over the data-generating processes from which the data are sampled. The data-generating processes considered in this paper include coin flips and several variants of first-order Markov processes.

Critical strengths

None

Critical weaknesses

The research question could be quite compelling with sharper framing. At present, however, the contribution feels limited in novelty. In particular, the central question of the paper appears to have been studied previously in [1] for example with other papers already cited. Although the setup there differs somewhat, it would be important to cite this work and clarify how the present paper extends or differs from it.

I also feel the writing would benefit from substantial revision before the paper is ready for submission. In its current form, it is difficult to follow, which makes it hard to properly evaluate the main claims.

A few specific points:

  • The abstract is difficult to read and would benefit from a substantial rewrite. The paper main point only became clear to me once I reached the introduction, so it may help to bring some of that clarity forward into the abstract.
  • The caption of Figure 2 includes the phrase “these estimates await numerical verification.”

Given these issues, I seriously doubt the authors thoroughly checked their paper before submission, which raises concerns about the overall quality and rigor of the work.

[1] Mikulik, Vladimir, et al. "Meta-trained agents implement bayes-optimal agents." Advances in neural information processing systems 33 (2020): 18691-18703.

Questions for authors

The caption of Figure 2 includes the phrase “these estimates await numerical verification.” This is confusing for what is presented as a completed paper. If the verification has since been done, the caption should be updated; if not, it would be worth completing before submission (Clarification).

1 comment
AuthorsOpenReview ↗

Response to reviewer q9hP

We disagree that the issues identified justify the general conclusion that the work lacks rigor. Appendices D and E document an extensive validation protocol: checks against closed-form cases, independent integration refinements and sampling runs, explicit numerical-error thresholds, checks of probability normalization and causal predictions, and independent recomputation of reported statistics. Numerical error is assessed separately from variation across test sequences. Code, saved results and reproduction instructions are also available.

The Figure 2 caveat concerns two Bayesian estimates at t=128 for L=2 and L=3. Their verification has now been completed, and we will update the caption. The other three behavioral tests and the t=64 copying comparison in Table 1 do not depend on those two estimates.

Mikulik et al. is relevant prior work, and we will cite it. It studies LSTM meta-learners on conditionally independent prediction tasks and bandits. Our contribution concerns transformers on structured Markov sources with dependencies across contexts, together with four quantitative behavioral tests beyond matching prediction curves.

The review identifies no specific error in a derivation or experimental result. We would appreciate clarification of the technical grounds for the soundness assessment.

Reviewer_r68GOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

The paper performs experiments on a (structured) mixture of Markov chains and compares the loss-per-token-position of a transformer to that of a Bayes-optimal predictor. The results show a generally good agreement between the two predictions, though no other point of comparison is given (e.g. SSM, MLP).

Critical strengths

The paper builds on the ability to compute the Bayes-optimal predictor in Markov chain-LSA and shows that for a wide variety of tasks, transformers reach the same performance when they are not otherwise constrained. The experiments are extensive.

Critical weaknesses

The paper doesn't always make its points plain and instead adopts a declarative and descriptive tone. I believe the main claim of the paper is that transformers act as Bayes-optimal (BO) predictors after training, but the authors never fully explicitly commit to this claim with the appropriate qualifications.

In addition, the paper does not systematically investigate any of the axes that come to mind in modern DL, like model capacity, number of training samples, and training time. Previous works found that, on natural language, in the data-constrained regime, architecture largely doesn't matter for next token prediction [1]; a more complete contribution would address this question. Similarly, previous works (e.g. [2]) have studied sample complexity, context length, input dimension, and task diversity in ICL and compared two different informed-bayes predictors.

Relatedly, the term "learning curves" is usually used to refer to training, e.g. are transformers as sample efficient as BO predictors, but this is not the case here. Instead, it refers to the in-context performance; it is worth emphasizing this and even revising this choice of words.

I find that the current submission contributes relatively little to my understanding, as it provides examples where transformers reach BO performance given unlimited capacity, data, and compute. I don't find this particularly surprising.

[1] Cagnetta F, Raventós A, Ganguli S and Wyart M 2026 Deriving Neural Scaling Laws from the statistics of natural language Online: http://arxiv.org/abs/2602.07488

[2] Lu Y M, Letey M I, Zavatone-Veth J A, Maiti A and Pehlevan C 2024 Asymptotic theory of in-context learning by linear attention Online: http://arxiv.org/abs/2405.11751

Questions for authors
  • Can you add a short description of LSA and a Bayes-predictor calculation? I believe that would make the paper more self-contained.
  • What are the main takeaways the reader is supposed to take from the paper?
5 comments
AuthorsOpenReview ↗

Response to reviewer r68G

The review’s characterization overlooks two central elements of the study.

First, our contribution goes beyond showing that transformers approach Bayesian prediction. Section 4 tests quantitative predictions for source mismatch, source changes, unseen contexts, and the benefit of token order. These tests probe consequences of learning shared source structure that agreement on ordinary prediction curves alone does not establish.

Second, the experiments use finite, explicitly reported resources (Table 3). Table 6 compares two- and four-layer models at both 6.4 and 25.6 million training sequences, alongside larger models. Further matched-budget comparisons appear in Sections 3.2, 3.4 and 3.6. Thus the characterization as agreement given “unlimited capacity, data, and compute” does not reflect the experimental design. These controls do leave some effects of width, depth and optimization unresolved.

Comparisons with other architectures and a broader scaling study would address additional questions. Our conclusions concern quantitative predictions of transformer behavior, rather than architectural superiority.

LSA is defined in Section 2.1, with the Bayesian calculation derived in Appendix B.2. We can bring a short example into the main text and clarify that the plotted curves measure in-context prediction as observed tokens increase.

I thank the authors for taking the time to read my review and reply to some of the concerns. I find that the current scope of the contribution, as presented in the manuscript, is too narrow for me to recommend acceptance at this time.

AuthorsOpenReview ↗

The reviewer response was simply null. No explanation.

Reply to the reviewer

We thank the reviewer for the follow-up. We note that the reply does not identify which concern remains, which part of our response was found insufficient, or in what respect the scope is too narrow. We addressed each point of the original review in detail; the reply engages with none of them. A review that cannot be responded to cannot be used to improve a paper, and we are glad to answer any specific remaining question.

Note to the area chairs

We ask the area chairs to weigh this review in light of the exchange above. The reviewer’s final assessment restates the original recommendation without addressing our rebuttal, without naming an outstanding concern, and without saying what a sufficient scope would be. “Too narrow” is asserted, not argued. A review that cannot be responded to cannot be used to improve a paper, and we believe it should not be the basis for rejecting one. We ask that the decision rest on the substantive reviews and on the manuscript itself.

Reviewer_TXEzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether transformers learn structure shared across sequences well enough to predict like a Bayesian model that knows the source family and prior. This question is pretty interesting, especially when observations in one context improve prediction in another.

  • The experiments use synthetic sources with fresh parameters for each sequence: random coins, independent Markov chains combined into tokens, an unknown shared base distribution, and mixtures of source families.
  • Larger transformers approach the Bayesian reference, including when token encodings or the shared base distribution are hidden. The supplement adds continued-training controls, direct predictive comparisons for several sources, and detailed checks of the numerical reference.
  • The contribution is a controlled empirical study of structured prediction, including ablations to: source mismatch, source changes, unseen contexts, and removing token order. An account of what internal computations are happening is hypothesised.
Critical strengths
  • The transformer reacts to changes in the data nearly as much as Bayes predicts. In one test, tokens have two independently generated parts. The authors change the rule for generating one part or both parts during a sequence. Bayes predicts how much extra prediction error this should cause, including a doubling when both parts change. The transformer closely matches these predictions.
  • The hidden-encoding result has informative training controls. Table 6 compares shows that. extra training alone leaves substantial gaps for small models. Larger models come much closer to Bayes. This supports a model-capacity effect, although it does not isolate depth from width or optimization schedule.
  • The unseen-context comparison tests useful transfer. In the shared-base source, other contexts help before the current token has ever been continued; the uniform-base control removes that advantage. At source depth 8 and context length 64, the reported transformer–Bayes loss difference is ~0.01 bits, while token counts lose much more: ~0.2 bits to Bayes.
  • The reported numerical checks are comprehensive + writing is good.
Critical weaknesses

I view the main contribution as useful empirical progress on an established Bayesian view of pretraining. The remaining concerns are how clearly this advance is distinguished from prior work and how strongly the shared-base experiments establish agreement with Bayes.

  • The related-work comparison is incomplete - Cannella, Teh, Han and Polyanskiy's work Universal priors: solving empirical Bayes via Bayesian inference and pretraining (2026) connects synthetic pretraining to hierarchical Bayes and directly compares transformer outputs with Bayesian estimates for Poisson data. The authors should discuss how the new results compare and contrast with the findings in this paper (also I am not sure whether this would count as concurrent work - the paper I share seems to have been Arxived in February 2026).
  • For the models trained only on shared-base sources, Tables 7–8 compare average loss at a fixed Zipf exponent of 1, although training covers a range of exponents. Similar loss on this slice need not mean similar next-token probabilities. Figure 6 directly checks the probability vectors for two other source families; I would like to see that check here too. The pooled-model experiments already test the full prior, but they use different models and do not resolve this question for the dedicated shared-base models.
  • The explanation of the size and training effects should remain tentative. Table 6 shows that extra training does not close the smaller models' gaps over the budgets tested. It does not distinguish a limits on what models can represent from optimizing difficulties to learn these predictor, or establish an inverse relationship between gap and training-set size. Appendix C labels its mechanisms as hypotheses and calls for a dedicated scaling study - I would apply that caution to its stronger statements about capacity floors and training rates in C.2 as well.
Questions for authors

The main remaining question is whether the models trained only on shared-base sources also match Bayes's predictions across their training prior. This would strengthen the interpretation of their small loss gaps without requiring a claim about internal Bayesian computation. How close are the dedicated shared-base models' next-token probability vectors to Bayes when test sources are drawn from the full training prior?

1 comment
AuthorsOpenReview ↗

Response to reviewer TXEz

Thank you for the careful and constructive review.

Our contribution is the quantitative study of structured sequential prediction, including four behavioral tests: source mismatch, source changes, unseen contexts, and the benefit of token order. Cannella et al. (2026) provide relevant complementary work on hierarchical Bayesian estimation for Poisson observations, including direct transformer–Bayes comparisons. We will add this reference and explain the connection and distinction.

The dedicated shared-base experiments demonstrate close agreement in expected loss across the tested depths and context lengths at α=1. Full-prior agreement of these models’ probability vectors is not established by the reported experiments. Equation (4) provides the appropriate distribution-level criterion under full-prior evaluation; the pooled-model experiments address a different predictor. We will make these distinctions explicit.

The continued-training controls show persistent gaps for smaller models and substantially closer agreement for larger models within the tested budgets. This is meaningful evidence about the effects of model size and training, while leaving representational limits and asymptotic training rates unresolved. We will revise C.2 to distinguish these empirical findings from the proposed explanation, and qualify the statements about capacity floors and inverse training-size scaling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.