acceptodds
Under review as a conference paper at ICLR 2027

In-context learning before specialisation: learning theory in a solvable topic model

Abstract

Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of *why* one solution might be favoured over another. Here we provide such a theory in a solvable topic model—documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.

Primary area:
learning theory
Keywords:
In-Context learning, Large Language Models, Learning Theory, Physics of Learning, Latent variable models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QWxkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the competition between in-context learning (ICL) and specialisation in sequence models, aiming to explain why different prediction strategies emerge at different stages of learning. Using an analytically tractable random topic model, the authors characterise two distinct predictors: a specialised predictor that exploits the finite set of training topics, and an in-context predictor that adapts to unseen topics using contextual token statistics. The paper derives an asymptotic learning curve for specialisation in a Bayesian learning setting, with the average excess loss scaling as (D(K-1)/(2NT)) under appropriate assumptions, and develops a sample-complexity argument predicting when ICL emerges before specialisation. Experiments with trained transformers across varying vocabulary sizes, topic diversities, sequence lengths, and training-set sizes show strong agreement with the predicted scaling relationships. The results provide a statistical explanation for how ICL can emerge before specialisation and subsequently decline as training data increases in sufficiently diverse task mixtures.

Critical strengths
  • The paper addresses an important question concerning the emergence of different generalisation strategies. Its main contribution is to connect the competition between ICL and specialisation to their distinct statistical learning complexities. The information-theoretic derivation of the specialisation learning curve (Section 3), together with the analysis of ICL emergence (Section 4), provides a coherent framework that goes beyond phenomenological descriptions. I consider this conceptual and theoretical contribution a strong reason for acceptance.

  • The empirical results provide convincing support for the proposed framework. In particular, Figures 3–5 demonstrate consistent scaling behaviour across different problem configurations, including the collapse of specialisation learning curves under the predicted rescaling and the emergence of ICL at a largely task-diversity-independent scale in the high-diversity regime. The agreement between the predicted scaling relationships and the empirical results, without fitting additional constants to the theoretical rescaling, is particularly compelling. The supplementary experiments further strengthen the evidence by examining different sequence lengths and prediction contexts.

Critical weaknesses

Tokens are i.i.d. within a document, so the in-context solution reduces to token counting, which is unusually cheap to learn. Topics are also uniform and equally frequent. Both features may favour the "ICL first" ordering, so the scope of the main claim needs to be stated more carefully. These theoretical limitations concerning the ICL emergence prediction are worth clarifying but do not fundamentally undermine the paper's main contributions.

Questions for authors
  • The ICL sample-complexity prediction in Eq. (13) is motivated by a count-linear predictor, while Appendix E notes that this family is generally misspecified on the finite-(D) training distribution. Could the authors further clarify the conditions under which the finite-(D) misspecification remains sufficiently small for the predicted ICL sample-complexity scaling to remain valid? This would help establish the scope of the proposed mechanism.

  • How sensitive are the conclusions to the topic model? (a) With long-tailed topic frequencies instead of uniform ones, would the crossover still be sharp? (b) Since only the specialisation scale grows with D, does the framework predict a reversed ordering at small D, and is this observed? (c) Would the ordering hold when the in-context rule is harder to learn, e.g. for mixtures of Markov chains? Answers to (a) and (b) would affect how broadly I read the main result; (c) can be treated as future work.

Reviewer_skCjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies when a learner trained on a finite dictionary of topics first acquires an in-context prediction rule and subsequently specialises to that dictionary. It derives Bayesian reference predictors, establishes an asymptotic excess-risk law for specialisation, and proposes a separate sample scale for learning the in-context predictor. Transformer experiments show learning curves consistent with the proposed scaling picture.

Critical strengths
  • The controlled setting makes the distinction between adaptation to unseen topics and specialisation to the training dictionary precise. The mutual-information interpretation in Eqs. (8)–(9) and the asymptotic analysis in Appendix D provide a useful theoretical account of the cost of learning the dictionary.
  • The experiments systematically vary vocabulary size, topic diversity and corpus size, with additional document-length experiments in Appendix H. Reporting both ID and OOD behaviour helps distinguish the two prediction strategies.
Critical weaknesses
  • The Bayesian learner in Section 3 knows the Dirichlet prior and therefore already implements the add-one predictor when no training corpus is available. Section 4 instead considers learning a general count-linear mapping with approximately K2K^2 parameters. Comparing these constructions suggests a possible explanation for strategy competition, but does not establish how a single learner acquires and selects between the strategies. Moreover, Appendix E explicitly acknowledges that the count-linear family is misspecified for the finite-DD training distribution. Its optimum need not be the add-one rule, and its estimation penalty involves tr⁡(JH−1)\operatorname{tr}(JH^{-1}), rather than automatically K2K^2. The applicability of the proposed ICL acquisition cost to the experimental regime therefore needs stronger justification. Equation (10) is derived for fixed D,K,TD,K,T as N→∞N\to\infty. It establishes a large-sample excess-risk coefficient, but it does not, by itself, locate the onset of specialisation at NT∼DKNT\sim DK. Appendix D’s fixed-position analysis also requires additional conditions concerning topic coverage and resolution. The empirical collapses support the proposed scale, but the manuscript should distinguish this evidence from a general finite-sample guarantee, particularly when using Nspec=DK/TN_{\mathrm{spec}}=DK/T to define the ICL window.

  • Figures 3–5 show informative curve collapses, but these do not establish that the transformer learns through the count-linear family analysed in Appendix E. Fitting that family on the same finite-DD corpora would directly test the proposed estimation cost. A learned mixture-of-multinomials baseline would similarly test specialisation. The claim of agreement with the Bayesian learning curve would also benefit from checking the coefficient in Eq. (10), beyond an inverse-power guide. An autoregressive neural topic-model comparison would provide useful additional evidence about architectural generality, although the matched baselines above are more central to the argument.

Questions for authors
  • Does the measured excess satisfy $ \frac{2NT\bigl(L_{\mathrm{AR}}-L_{\mathrm{AR}}[q]\bigr)} {D(K-1)} \longrightarrow 1 $ in the regime where Eq. (10) is expected to apply? How is the empirical crossover defined and compared with its proposed scale?

  • Are selected checkpoints evaluated on independent ID and OOD test sets? Appendix G states that validation losses enter the learning curves, and Figure 5 selects checkpoints using OOD validation loss. Please clarify the separation between model selection and final evaluation, including the sampling of OOD topics.

Reviewer_WK1oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors points out that the sequence model's stratergies determine how the model behances beyound training distribution. And recent works shows many different strategies theoretically, and empirically. However, there is no learning theory, hence no fundamental understanding on why one solution favours over the other. Hence the author provides a theory in a solvable topic model. The paper also contributes a statistical mechanism for in-context phase, a parameter-free agreement with trained transformers, and a separation from classical spectral methods.

Critical strengths
  1. The paper is generally structured and well written, with one suggestion is that the figure 1 can be in second page, so that the reader sees the introduction first.

  2. The authors analyse two distinct Bayes-optimal predictors. One relying on knowledge of the DD topics in data generation process and second, an in-context one, equivalent to the add-one estimator from statistics under data model. Also, the figure 2 supports the idea that high topic diversity encourages in-context learning, while low diversity encourages specialization in familiar topics.

  3. The author shows that the transformers approach the specialised predictor at the rate predicted by the Bayesian learning theory (Figure 3).

Critical weaknesses
  1. For the experiment a one-block, single-head transformer is used. Testing with deeper models, multiple heads, and different training settings would help assess the robustness of the findings.

  2. Another limitation is that model size and training settings are tuned separately for each condition (Appendix G, Hyperparameter selection). A comparison using a fixed architecture and training setup would help establish how robust the observed scaling pattern is.

  3. The comparison with spectral methods uses different sampling units and focus on subspace recovery rather than prediction performance.

Questions for authors
  1. It will be good to add main takeaway point in figure rather than explaining about the details of the figure (as in figure 2).

  2. The extra space in page 9 could have been utilized with some results.

  3. Figure 2 shows transformer results for (D=32,N=16384)(D=32, N=16384) and (D=8192,N=65536)(D=8192, N=65536). Including other values of DD and NN would help clarify the transition between specialisation and in-context learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.