- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies in-content learning across different modalities and attempts to find a common structure within ICL across domains. Basically the paper tries to answer the following questions:
- Does ICL emerge in different modalities?
- Does ICL in different modalities share certain capabilities?
- How does model scale affect ICL?
To answer these questions, they setup a controlled framework where they define abstract tasks using binary digits and translate the same abstract tasks to different modalities including language, genome, integer sequences, time series, images, and proteins. Results show that in some domains, ICL emerges and the ICL capabilities are highly correlated.
Critical strengths
This paper studies two interesting questions in ICL 1) Does ICL emerge in different modalities? and 2) Does ICL in different modalities share certain capabilities and provide a systematic approach and great insights with clear discussions.
Authors transparently discuss the limitations of their methodology and attempt to address those.
They expand the method of Breslow, Nathan, et al. to other domains allowing them to capture the shared capabilities across domains which the method of expansion to the best of my knowledge is a novel approach.
Critical weaknesses
- Main concern This paper is mostly the expansion of Breslow, Nathan, et al. in new domains such as music, chess, time-series, etc. Authors do show that in some domains such as chess and music the ICL capability does not emerge. However, they do not provide insight on why we see these capabilities in some domains but not in the other.
Similarly, the correlation between the image task and other language task was seen to be very low. However, no insight is provided on why that is the case and how the image and non-image learned ICL capabilities differ from each other. Basically, what kind of ICL do we have in image models that we do not have in other tasks. Is there any way to design experiments or do analysis on the task vectors to answer this question?
- My second concern is about the choice of models. Some of these models such as Qwen are trained large amount of unknown datasets that might violate the second condition of the authors: "the pretraining data is naturally occurring rather than explicitly crafted to elicit ICL.". Overall, paper needs to acknowledge that without careful control of the training setup of the models, or using more than one model per domain, the results might not be conclusive.
Questions for authors
Please see above.
3 comments
(cont.)
“My second concern is about the choice of models. Some of these models such as Qwen are trained large amount of unknown datasets that might violate the second condition of the authors: "the pretraining data is naturally occurring rather than explicitly crafted to elicit ICL.". Overall, paper needs to acknowledge that without careful control of the training setup of the models, or using more than one model per domain, the results might not be conclusive.”
We agree - and because of this, we add an explanation in our “Resolving relative ICL strength across and within modalities” subsection that explains how we address this in the case of Qwen - namely, using a randomized digit encoding instead of a literal-bitstring one.
We explore the impact of this choice in the newly added Appendix N, where we see Qwen’s accuracy drop from 75.8% at 64 shots with a literal bitstring encoding to 30% with our randomized digit one. Naturally, this result doesn’t guarantee that we eliminated all contamination from Qwen’s pretraining data - but it does show that our choice of encoding resolves a substantial amount of it.
We likewise test a different language model - Mistral 7B v0.3 - in Appendix M, to see if it exhibits a similar profile to Qwen. Though the absolute magnitude of its ICL gains are smaller (5.7pp versus Qwen3 14B’s 13.9pp), it is fairly correlated with Qwen3 14B at 64 shots (0.66), especially relative to Qwen3 14B’s average correlation with other modalities (0.49). Finally, we acknowledge many more caveats in our “Resolving relative ICL strength across and within modalities” section than we did previously - for instance, we emphasize: “It is difficult, if not impossible, to disentangle the effect of the model’s priors from its ability to learn in-context.” and “Even within a family, scaling comparisons remain confounded when checkpoints differ in pretraining data or architecture.” We think these caveats better contextualize the scope of our results and address your concern.
We thank the reviewer for your feedback, especially the idea to test more than one model per domain and to better address prior contamination in language. Your feedback has made our findings more robust. For ease of reviewing, all revisions in our updated draft are highlighted in blue.
I thank the authors for taking the time to respond to my comments. Some of my concerns have been addressed. I maintain my positive score.
We thank the reviewer for recognizing our transparency, our expansion of Breslow et al.’s method, and that our paper addresses interesting questions by investigating ICL in different modalities. We address your concerns below:
We agree that we expand on Breslow et al. and test their framework and tasks in new domains. However, we also introduce a variety of novel techniques that distinguish our work. First, we define ICL emergence by comparing clean accuracy with a deranged control across modalities and shot counts, something Breslow et al. only does at one shot count for one genomic task in their Appendix E. Second, we introduce the concept of correlating ICL gain across modalities, and the use of permutation tests to verify the significance of correlation. Finally, we enhance Breslow et al.’s methodology with a variety of smaller improvements - an upper bound on ambiguity probability, Holm adjusted p-values, semantic task clustering (Appendix J), and T=128 trials per task instead of T=8 for tighter per-task analysis. We think these constitute a meaningfully novel extension of the work established in Breslow et al..
We agree with your concern. Therefore, we have greatly extended our subsection “Resolving relative ICL strength across and within modalities.” with a cursory overview of the tasks image models excel on (bit reversal) and a tentative explanation for why (pretraining augmentation or symmetry priors), along with analysis of training priors for NextTerm and Qwen3 (which we discuss at length in response to your secondary concern). We also note that Figure 5 shows model performance per semantic task cluster, and clearly demonstrates ImageGPT-large’s outsized bit reversal, local shift, and half swap abilities. Finally, we make a preliminary investigation into task vectors in Appendix P - and compare the correlation of task vector similarity matrices across modalities. Image remains less correlated with the other modalities (which again form a block), and we observe that NextTerm is entirely an outlier - preliminary analysis suggests this is because NextTerm’s task vectors correlate with whether or not NextTerm successfully learned the task in context rather than the semantic structure of the task itself.
(cont. in next comment)