- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
This paper investigates the competition between task retrieval and task learning in Transformer in-context learning using finite mixtures of Markov chains. It theoretically characterizes the finite-prior Bayes predictor, identifying the ratio of prompt length to the number of pretraining tasks as a key factor governing retrieval–learning transitions. It establishes quantitative error bounds for both regimes and provide conditional guarantees that pretrained Transformers approximate Bayesian predictions when is sufficiently small. Experiments further illustrates that Transformers increasingly deviate from finite-prior Bayes as grows, exhibiting a stronger relative preference for task learning.
Critical strengths
-
I found the perspective that the ratio governs both retrieval and learning to be interesting, as it provides explicit and quantitative theoretical guarantees for Bayes predictors under finite priors.
-
The paper performs experiments investigating the relationship between trained Transformers and Bayes predictors. In particular, the observation that the Transformer diverges from the Bayes prediction as increases is quite interesting.
Critical weaknesses
- Limited theoretical characterization of Transformer behavior.
The theoretical guarantees for Transformer retrieval and phase diagrams remain unestablished. Most of the results in this paper offered few surprises beyond those of Raventós et al. or Park et al.. The paper left the most interesting Transformer-specific phenomena theoretically unexplained.
- Generality of the scaling.
I am skeptical about the generality of the scaling. Intuitively, in the presence of -dimensional unknown variables, I would expect an order like (meaning holds simply because in this specific case). How should we interpret or understand this scaling?
Raventós et al., Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression, NeurIPS 2023.
Park et al., Competition Dynamics Shape Algorithmic Phases of In-Context Learning, ICLR 2025.
Questions for authors
-
To improve presentation and soundness scores (as well as rate), could you clearly organize what has been theoretically shown (and to what extent) versus what has been empirically illustrated? A table may be helpful.
-
In the pre-training section, it is not proven that the trained Transformer sufficiently approaches the Bayes optimal solution. While this may not be necessary for the present study, shouldn't prior works on statistical theory and the approximation of Bayes predictors/conditional distributions, such as Wakayama & Suzuki or Liu etal., be cited respectfully?
Liu et al. (2025). In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimality. arXiv:2508.08673. Wakayama & Suzuki (2026). In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning. ICML 2026.