- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the competition between in-context learning (ICL) and specialisation in sequence models, aiming to explain why different prediction strategies emerge at different stages of learning. Using an analytically tractable random topic model, the authors characterise two distinct predictors: a specialised predictor that exploits the finite set of training topics, and an in-context predictor that adapts to unseen topics using contextual token statistics. The paper derives an asymptotic learning curve for specialisation in a Bayesian learning setting, with the average excess loss scaling as (D(K-1)/(2NT)) under appropriate assumptions, and develops a sample-complexity argument predicting when ICL emerges before specialisation. Experiments with trained transformers across varying vocabulary sizes, topic diversities, sequence lengths, and training-set sizes show strong agreement with the predicted scaling relationships. The results provide a statistical explanation for how ICL can emerge before specialisation and subsequently decline as training data increases in sufficiently diverse task mixtures.
Critical strengths
-
The paper addresses an important question concerning the emergence of different generalisation strategies. Its main contribution is to connect the competition between ICL and specialisation to their distinct statistical learning complexities. The information-theoretic derivation of the specialisation learning curve (Section 3), together with the analysis of ICL emergence (Section 4), provides a coherent framework that goes beyond phenomenological descriptions. I consider this conceptual and theoretical contribution a strong reason for acceptance.
-
The empirical results provide convincing support for the proposed framework. In particular, Figures 3–5 demonstrate consistent scaling behaviour across different problem configurations, including the collapse of specialisation learning curves under the predicted rescaling and the emergence of ICL at a largely task-diversity-independent scale in the high-diversity regime. The agreement between the predicted scaling relationships and the empirical results, without fitting additional constants to the theoretical rescaling, is particularly compelling. The supplementary experiments further strengthen the evidence by examining different sequence lengths and prediction contexts.
Critical weaknesses
Tokens are i.i.d. within a document, so the in-context solution reduces to token counting, which is unusually cheap to learn. Topics are also uniform and equally frequent. Both features may favour the "ICL first" ordering, so the scope of the main claim needs to be stated more carefully. These theoretical limitations concerning the ICL emergence prediction are worth clarifying but do not fundamentally undermine the paper's main contributions.
Questions for authors
-
The ICL sample-complexity prediction in Eq. (13) is motivated by a count-linear predictor, while Appendix E notes that this family is generally misspecified on the finite-(D) training distribution. Could the authors further clarify the conditions under which the finite-(D) misspecification remains sufficiently small for the predicted ICL sample-complexity scaling to remain valid? This would help establish the scope of the proposed mechanism.
-
How sensitive are the conclusions to the topic model? (a) With long-tailed topic frequencies instead of uniform ones, would the crossover still be sharp? (b) Since only the specialisation scale grows with D, does the framework predict a reversed ordering at small D, and is this observed? (c) Would the ordering hold when the in-context rule is harder to learn, e.g. for mixtures of Markov chains? Answers to (a) and (b) would affect how broadly I read the main result; (c) can be treated as future work.