- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the relationship between delayed generalization and epistemic uncertainty in ICL. Predictive uncertainty is estimated using IVON and Last-Layer Laplace on a modular arithmetic benchmark with latent linear tasks. Experimental results reveal that models generalization ability to new tasks exhibit lower epistemic uncertainty. Moreover, uncertainty during training does not decrease monotonically, and namely, it first increases to a peak before declining. Furthermore, using a simplified Bayesian linear model, this work theoretically demonstrates that slow-learning eigenmodes in the covariance matrix of training data can simultaneously give rise to both delayed generalization and non-monotonic dynamics in uncertainty.
Critical strengths
I found it interesting how observing changes in epistemic uncertainty could help us understand how models acquire generalization ability. Also, I believe that the fact that the improvement in performance and the change in uncertainty (self-assessment) don't have a simple monotonic relationship suggests an intriguing parallel between human metacognition and AI uncertainty estimation.
Critical weaknesses
The task in Eq. (1) seems quite simple. Once two in-context examples are provided, the model should be able to solve the subsequent pairs and beyond. So, I have a concern that the problem setting might be very toy and a question regarding whether pretraining is sufficient.
Questions for authors
In relation to the weakness mentioned above, could you share any results regarding sufficient training convergence? To the best of my knowledge, even in toy model setups, the emergence of ICL requires a fairly large sample size (and number of training steps). For instance, Garg et al. (2022) and many subsequent studies adopt massive online training even when working with toy models.
I would appreciate it and reconsider the soundness score and the rate if you could share your thoughts on this aspect.
Garg et al. (2022), What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, NeurIPS 2022.
4 comments
Response pt2
Q1: Comparison with Garg et al. (2022)
Garg et al. [2] train their standard linear-regression model using fresh functions and inputs sampled from Gaussian distributions in 20 dimensions. Their schedule uses 500,000 updates with 64 prompts per batch, totaling prompts. Our training volume is of the same order, while our experiment explicitly controls the diversity of a finite task family with possible task vectors. We reserve task vectors for evaluation and construct training sequences from the designated training tasks and inputs. This separation lets us study how finite pretraining diversity affects subsequent inference of unseen tasks from context.
Garg et al. also examine training with a finite set of functions: their Section 6 reports performance close to their unrestricted model with 10,000 distinct linear functions. Thus, finite task diversity is itself an informative experimental variable for studying the emergence of ICL. In our benchmark, varying this quantity exposes the generalization regimes whose uncertainty dynamics we analyze.
We hope these clarifications resolve the reviewer's concerns, and we remain available for any further discussion.
References
[1] Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to Grok: Emergence of In-Context Learning and Skill Composition in Modular Arithmetic Tasks. NeurIPS, 2024.
[2] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. NeurIPS, 2022.
Feedback
Dear Reviewer,
We would appreciate hearing whether our response addressed your questions, in particular on task simplicity, training sufficiency (context-length results and new training curves in Figure 14) and the comparison with Garg et al. (2022).
If any concern remains, please let us know while there is time to discuss it. Otherwise, we would be grateful if you would consider updating your rating, as you previously kindly suggested.
Thank you for your time.
Thank you for your response. Your rebuttal resolved most of my concerns. In particular, comparison with Garg et al. makes sense. I will keep my positive evaluation.
Response
We thank the reviewer for the constructive comments and for highlighting the relationship between generalization and uncertainty estimation.
W1: Task identifiability and learning from context
The reviewer is correct that two examples identify the latent task (a,b) when their input pairs are linearly independent modulo 29. For inputs (x1,y1) and (x2,y2), this condition is x1y2−x2y1≡0(mod29). An exact modular linear solver can then recover (a,b) and predict every subsequent query.
The transformer learns its prediction procedure from sequences of numeric tokens. Our context-length experiments measure how effectively this learned procedure uses the available examples. At fixed weights, OOD-Val accuracy remains low with at most two examples and increases as more examples are provided (Figures 5, 8, and 9). For IVON at (T=256,α=0.8), median accuracy across the three evaluated seeds increases from 0.033 with no examples to 0.871 with 31 examples. The same trained models therefore exhibit substantial held-out-task generalization when given sufficient context.
He et al. (2024) [1] provide a relevant mechanistic account on this benchmark. With few examples, their two- and four-layer models use Ratio Matching, which succeeds when the query input is a modular multiple of an input in the context. With more examples, their four-layer models combine inputs through Modular Regression and achieve much higher OOD accuracy. This analysis illustrates how the learned procedure can require more context than an exact solver. Our experiments characterize the accompanying changes in epistemic uncertainty (EU) in six-layer models; identifying the precise algorithm implemented by these models remains a separate mechanistic question.
W1 & Q1: Training duration and evidence of ICL
Appendix E specifies a six-layer transformer, modulus p=29, batches of 1024 sequences, and K=32 triplets per sequence, corresponding to 31 in-context examples and one query. Training supervises all answer positions and uses task-balanced batches with the parallelogram task construction of He et al. We vary the effective number of training tasks over T∈{32,64,128,256,512} and input coverage over α∈{0.3,0.4,0.5,0.6,0.7,0.8}, with four seeds per configuration. The default horizon is 100,000 updates, extended to 200,000 for T=512; the latter matches the training horizon in He et al. At the nominal batch size, these schedules correspond to approximately 1.02×108 and 2.05×108 sequence presentations. These counts include repeated exposure to the finite training tasks and inputs.
The reported schedules are sufficient for ICL to emerge in the generalizing configurations. Across the 120 IVON runs, 37 achieve stable generalization to unseen tasks on familiar inputs only, and nine also achieve stable generalization when both tasks and inputs are held out. Here, stable generalization means a median accuracy of at least 0.75 over the final five evaluation checkpoints (Appendix A.1). The fixed-weight context-length results above further demonstrate that the trained models use additional in-context evidence to predict held-out tasks.
Figure 14 (new, Appendix A.4) reports training loss and accuracy curves for all configurations of the MAP (AdamW) runs, and shows how the training horizon follows the dynamics of each regime. For T≤128, the training loss drops to near zero within the first few thousand updates, while accuracy on held-out tasks remains near chance for the entire horizon: these runs fit their training set and show no transition to held-out tasks. For T=256 with α≥0.5, accuracy on all splits rises sharply and plateaus within the 100,000-update horizon. For T=512, transitions to OOD generalization unfold more slowly; for computational reasons, we therefore reserve the longer 200,000-update horizon, which matches He et al., for this largest task count. All nine runs used in our peak-order analysis (Table 3) come from configurations whose accuracy plateaus during training (T=256 with α∈{0.6,0.7,0.8} and T=512 with α=0.8).
He et al. also report that OOD generalization can emerge and subsequently deteriorate during training in six-layer models [1]. This motivates examining the full accuracy and uncertainty trajectories alongside final performance, as we do in Section 3.3.