acceptodds
Under review as a conference paper at ICLR 2027

A Bayesian Perspective on Task Retrieval and Task Learning in In-Context Learning on Markovian Data

Abstract

In-context learning enables pretrained transformers to adapt from a prompt without parameter updates, but it remains unclear when this behavior arises from retrieving memorized pretraining tasks (i.e., task retrieval) versus learning a rule from in-context data (i.e., task learning). We study this question through a finite-mixture Markov data model, where each latent Markov chain represents a pretraining task. Our analysis identifies the ratio of prompt length to the number of pretraining tasks as a key factor. We first establish quantitative guarantees showing that the finite-prior Bayesian predictor associated with the pretraining tasks favors retrieval at large ratios and learning from the prompt at sufficiently small ratios. We further show that, at small ratios, well-trained transformers make predictions close to those of the finite-prior Bayes predictor. At larger ratios, our experiments show increasing deviations from Bayes, with transformers exhibiting a stronger relative preference for learning. Our findings clarify how prompt length and pretraining diversity jointly shape in-context learning and when Bayesian prediction explains transformer behavior.

Primary area:
optimization
Keywords:
transformers, in-context learning, task retrieval, task learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MwwCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper investigates the competition between task retrieval and task learning in Transformer in-context learning using finite mixtures of Markov chains. It theoretically characterizes the finite-prior Bayes predictor, identifying the ratio of prompt length nn to the number of pretraining tasks KK as a key factor governing retrieval–learning transitions. It establishes quantitative error bounds for both regimes and provide conditional guarantees that pretrained Transformers approximate Bayesian predictions when n/Kn/K is sufficiently small. Experiments further illustrates that Transformers increasingly deviate from finite-prior Bayes as n/Kn/K grows, exhibiting a stronger relative preference for task learning.

Critical strengths
  1. I found the perspective that the ratio n/Kn/K governs both retrieval and learning to be interesting, as it provides explicit and quantitative theoretical guarantees for Bayes predictors under finite priors.

  2. The paper performs experiments investigating the relationship between trained Transformers and Bayes predictors. In particular, the observation that the Transformer diverges from the Bayes prediction as n/Kn/K increases is quite interesting.

Critical weaknesses
  1. Limited theoretical characterization of Transformer behavior.

The theoretical guarantees for Transformer retrieval and phase diagrams remain unestablished. Most of the results in this paper offered few surprises beyond those of Raventós et al. or Park et al.. The paper left the most interesting Transformer-specific phenomena theoretically unexplained.

  1. Generality of the n/Kn/K scaling.

I am skeptical about the generality of the n/Kn/K scaling. Intuitively, in the presence of dd-dimensional unknown variables, I would expect an order like nd/2/Kn^{d/2}/K (meaning n/Kn/K holds simply because d=2d = 2 in this specific case). How should we interpret or understand this n/Kn/K scaling?


Raventós et al., Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression, NeurIPS 2023.

Park et al., Competition Dynamics Shape Algorithmic Phases of In-Context Learning, ICLR 2025.

Questions for authors
  • To improve presentation and soundness scores (as well as rate), could you clearly organize what has been theoretically shown (and to what extent) versus what has been empirically illustrated? A table may be helpful.

  • In the pre-training section, it is not proven that the trained Transformer sufficiently approaches the Bayes optimal solution. While this may not be necessary for the present study, shouldn't prior works on statistical theory and the approximation of Bayes predictors/conditional distributions, such as Wakayama & Suzuki or Liu etal., be cited respectfully?


Liu et al. (2025). In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimality. arXiv:2508.08673. Wakayama & Suzuki (2026). In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning. ICML 2026.

Reviewer_o3SyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how context length n and pretraining task diversity K influence retrieval and learning in ICL on Markov data. It provides two main theorems. Theorem 3.3 gives finite-sample bounds characterizing the behavior of the finite-prior Bayes predictor relative to the retrieval benchmark and the uniform-prior Bayes predictor. Theorem 3.4 connects pretraining excess risk to Transformer prediction errors on unseen tasks, measured relative to these Bayesian predictors. Experiments examine the predicted scaling laws and deviations of trained Transformers from Bayesian predictions.

Critical strengths

The underlying analysis addresses a technically nontrivial and challenging problem.

  • Theorem 3.3 characterizes how the finite-prior Bayes predictor depends on the ratio n/K in finite samples.
  • Theorem 3.4 provides a clean decomposition separating finite-library approximation error from pretraining excess risk epsilon_K,n when evaluating on unseen tasks.
  • Figure 5 directly tests the finite-prior Bayes errors across several (K, n) settings. The results support the scaling forms derived in Theorem 3.3 and suggest that they describe the observed errors well in the tested regimes.
Critical weaknesses

Weaknesses and Questions:

The paper only provides direct quantitative comparisons with the theoretical bounds in Appendix Figure 5 (for the Bayes predictor). In all main figures (Figures 1, 3, and 4), the study is largely empirical, and the theory provides few quantitative predictions to compare directly with the Transformer numerics.

  1. Limited quantitative predictions for actual Transformer behavior: In Figure 3, the theory appears to match the pretrained Transformer primarily when n ~ 10, while the Transformer shows clear deviations from the finite-prior Bayes predictor in the regime where n, K >> 1 and n/K < 1. Can the current theoretical framework explain this deviation? Since n, K >> 1 is an important regime in which most pretrained Transformer models operate, any discussion or analysis would be very helpful.

  2. Missing empirical results for Eq. (13) in Theorem 3.4: The paper does not provide numerical results to directly verify Eq. (13). Specifically, although Section 4.2 mentions estimating the pretraining excess risk epsilon_(K,n), those empirical values are missing from the paper. Then it becomes difficult to tell whether the changes across the curves in Figure 4a are driven by poor prompt coverage at larger n/K or by variations in the Transformer's in-distribution optimization error epsilon_(K,n). The paper also does not empirically test the second bound in Eq. (13) for RMSE_n(f_hat, f_unif*). More numerical comparisons are necessary to support Theorem 3.4.

  3. Ambiguities and inconsistencies across Figures 3 and 4:

  • Unexplained changes in metrics: The evaluation metric shifts from the L1-based alignment score s(K,n) in Figure 3 to the MSE-based preference shift Delta_pref in Figure 4b. Why are two different metrics used, and is this choice due to a difference in the underlying task distributions or theoretical deviations?

  • Unspecified task distribution in Figure 3: Section 4.1 does not specify whether the evaluation tasks in Figure 3 come from the in-distribution training library or unseen out-of-distribution tasks. It would be helpful to explicitly label and distinguish in-distribution and out-of-distribution task evaluations throughout the paper.

Questions for authors

Please refer to Weaknesses section.

Reviewer_y6EyOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies task retrieval vs task learning on a simple setting (binary Markov chains). It studies how the ratio between sequence length and tasks in the training distribution govern the change of regime. It then illustrates its theoretical findings on actual experiments.

Critical strengths
  • Quantifies and empirically validates the influence of the aforementioned ratio
Critical weaknesses

My main concern is the informativeness of the paper: ICL has been thoroughly studied in the previous years and it is not clear how much new insight this paper brings to the table. In particular, it is not clear to me how "new" are Theorems 3.3 and 3.4

Another concern is the absence of consideration case: an important ICL question is whether the learned model can generalize to geenuinely new and OOD tasks, which this paper does not discuss.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.