acceptodds
Under review as a conference paper at ICLR 2027

Convergent Emergence of In-Context Learning Across Modalities

Abstract

Few-shot in-context learning (ICL), the capacity of a model to infer abstract patterns from input-output examples provided in its prompt and apply them to new inputs, has been extensively studied in large language models trained for next-token prediction on human text. Recently, few-shot ICL has been demonstrated in autoregressive genomic models as well. This raises a question: does ICL emerge broadly across domains, and if so, what common structure is shared? To address both, we develop a controlled cross-modality framework that instantiates the same task suite in a variety of modalities to test what we call the Convergent Emergence Hypothesis: the idea that few-shot ICL, when it emerges, shares a common cross-modality difficulty profile - i.e., tasks that benefit from ICL in one modality tend to benefit in others. We show that paired-mapping ICL emerges across six modalities (language, genome, integer sequences, time series, images, and proteins), surpasses controlled baselines, and has correlated per-task effects across five of them. Together, these results provide support for the Convergent Emergence Hypothesis in some modalities, but not all.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, next-token prediction, emergent capabilities, cross-modality evaluation, foundation models

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_swxEOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies in-content learning across different modalities and attempts to find a common structure within ICL across domains. Basically the paper tries to answer the following questions:

  1. Does ICL emerge in different modalities?
  2. Does ICL in different modalities share certain capabilities?
  3. How does model scale affect ICL?

To answer these questions, they setup a controlled framework where they define abstract tasks using binary digits and translate the same abstract tasks to different modalities including language, genome, integer sequences, time series, images, and proteins. Results show that in some domains, ICL emerges and the ICL capabilities are highly correlated.

Critical strengths

This paper studies two interesting questions in ICL 1) Does ICL emerge in different modalities? and 2) Does ICL in different modalities share certain capabilities and provide a systematic approach and great insights with clear discussions.

Authors transparently discuss the limitations of their methodology and attempt to address those.

They expand the method of Breslow, Nathan, et al. to other domains allowing them to capture the shared capabilities across domains which the method of expansion to the best of my knowledge is a novel approach.

Critical weaknesses
  1. Main concern This paper is mostly the expansion of Breslow, Nathan, et al. in new domains such as music, chess, time-series, etc. Authors do show that in some domains such as chess and music the ICL capability does not emerge. However, they do not provide insight on why we see these capabilities in some domains but not in the other.

Similarly, the correlation between the image task and other language task was seen to be very low. However, no insight is provided on why that is the case and how the image and non-image learned ICL capabilities differ from each other. Basically, what kind of ICL do we have in image models that we do not have in other tasks. Is there any way to design experiments or do analysis on the task vectors to answer this question?

  1. My second concern is about the choice of models. Some of these models such as Qwen are trained large amount of unknown datasets that might violate the second condition of the authors: "the pretraining data is naturally occurring rather than explicitly crafted to elicit ICL.". Overall, paper needs to acknowledge that without careful control of the training setup of the models, or using more than one model per domain, the results might not be conclusive.
Questions for authors

Please see above.

3 comments

We thank the reviewer for recognizing our transparency, our expansion of Breslow et al.’s method, and that our paper addresses interesting questions by investigating ICL in different modalities. We address your concerns below:

"Main concern: This paper is mostly the expansion of Breslow, Nathan, et al. in new domains such as music, chess, time-series, etc."

We agree that we expand on Breslow et al. and test their framework and tasks in new domains. However, we also introduce a variety of novel techniques that distinguish our work. First, we define ICL emergence by comparing clean accuracy with a deranged control across modalities and shot counts, something Breslow et al. only does at one shot count for one genomic task in their Appendix E. Second, we introduce the concept of correlating ICL gain across modalities, and the use of permutation tests to verify the significance of correlation. Finally, we enhance Breslow et al.’s methodology with a variety of smaller improvements - an upper bound on ambiguity probability, Holm adjusted p-values, semantic task clustering (Appendix J), and T=128 trials per task instead of T=8 for tighter per-task analysis. We think these constitute a meaningfully novel extension of the work established in Breslow et al..

"Authors do show that in some domains such as chess and music the ICL capability does not emerge. However, they do not provide insight on why we see these capabilities in some domains but not in the other. Similarly, the correlation between the image task and other language task was seen to be very low. However, no insight is provided on why that is the case and how the image and non-image learned ICL capabilities differ from each other. Basically, what kind of ICL do we have in image models that we do not have in other tasks. Is there any way to design experiments or do analysis on the task vectors to answer this question?"

We agree with your concern. Therefore, we have greatly extended our subsection “Resolving relative ICL strength across and within modalities.” with a cursory overview of the tasks image models excel on (bit reversal) and a tentative explanation for why (pretraining augmentation or symmetry priors), along with analysis of training priors for NextTerm and Qwen3 (which we discuss at length in response to your secondary concern). We also note that Figure 5 shows model performance per semantic task cluster, and clearly demonstrates ImageGPT-large’s outsized bit reversal, local shift, and half swap abilities. Finally, we make a preliminary investigation into task vectors in Appendix P - and compare the correlation of task vector similarity matrices across modalities. Image remains less correlated with the other modalities (which again form a block), and we observe that NextTerm is entirely an outlier - preliminary analysis suggests this is because NextTerm’s task vectors correlate with whether or not NextTerm successfully learned the task in context rather than the semantic structure of the task itself.

(cont. in next comment)

(cont.)

“My second concern is about the choice of models. Some of these models such as Qwen are trained large amount of unknown datasets that might violate the second condition of the authors: "the pretraining data is naturally occurring rather than explicitly crafted to elicit ICL.". Overall, paper needs to acknowledge that without careful control of the training setup of the models, or using more than one model per domain, the results might not be conclusive.”

We agree - and because of this, we add an explanation in our “Resolving relative ICL strength across and within modalities” subsection that explains how we address this in the case of Qwen - namely, using a randomized digit encoding instead of a literal-bitstring one.

We explore the impact of this choice in the newly added Appendix N, where we see Qwen’s accuracy drop from 75.8% at 64 shots with a literal bitstring encoding to 30% with our randomized digit one. Naturally, this result doesn’t guarantee that we eliminated all contamination from Qwen’s pretraining data - but it does show that our choice of encoding resolves a substantial amount of it.

We likewise test a different language model - Mistral 7B v0.3 - in Appendix M, to see if it exhibits a similar profile to Qwen. Though the absolute magnitude of its ICL gains are smaller (5.7pp versus Qwen3 14B’s 13.9pp), it is fairly correlated with Qwen3 14B at 64 shots (0.66), especially relative to Qwen3 14B’s average correlation with other modalities (0.49). Finally, we acknowledge many more caveats in our “Resolving relative ICL strength across and within modalities” section than we did previously - for instance, we emphasize: “It is difficult, if not impossible, to disentangle the effect of the model’s priors from its ability to learn in-context.” and “Even within a family, scaling comparisons remain confounded when checkpoints differ in pretraining data or architecture.” We think these caveats better contextualize the scope of our results and address your concern.

We thank the reviewer for your feedback, especially the idea to test more than one model per domain and to better address prior contamination in language. Your feedback has made our findings more robust. For ease of reviewing, all revisions in our updated draft are highlighted in blue.

I thank the authors for taking the time to respond to my comments. Some of my concerns have been addressed. I maintain my positive score.

Reviewer_R3z9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper explores whether few-shot in-context learning (ICL) emerges from next-token pretraining on non-linguistic data (such as images, time sequences, chess, etc.), and whether its per-task benefits are shared across modalities, which the authors refer to as the “Convergent Emergence Hypothesis”.

The authors create a single suite of 100 eight-bit bitstring transformations in six modalities (language, genome, integer sequences, images, time series, proteins) through modality-specific encodings and off-the-shelf pretrained models for each modality.

For each modality and task, they run several trials to ensure reliability of the results. Each task is run with a “clean” prompt and a “deranged” control in which demonstrated outputs are shuffled across inputs, so the clean-minus-deranged gap measures the benefit of the true ICL beyond trivial guessing.

The authors show that for their primary set of modalities, exact match accuracy rises with shot count while the deranged control stays flat in all six modalities. They also show that the per-task gap profiles are positively correlated across five of six modalities, with the exception of ImageGPT. They also show that that larger models are generally better at ICL. They also show that for models trained on chess and music modalities, there's little evidence of ICL emergence.

Critical strengths
  1. The paper presents a sound approach with a suite to reliably prove evidence of emergent ICL in a given modality, using insightful experiments on various modalities and models.
  2. The idea of using a label-shuffled control, per-trial resampling of mappings, associated mathematical proofs behind their assumptions are all signs of a strong theoretical contribution. The framework is also easy to apply to other modalities using the bitstring format.
  3. The authors call out the negative results (for chess, music) and also clarify that their assumptions of "natural data" are violated by some of the models, which is a great aspect of this paper. The authors evaluate their claims holistically and present both positive and negative results.
Critical weaknesses
  1. The task domain is narrow, so generality is limited. All tasks chosen by the authors are 8-bit transformatoins, many solvable by copying or shifting, so the profiles may reflect simple behaviors rather than ICL in general. The authors could explore either narrowing the claims or add a non-bitstring task family.

  2. The correlation is a weak evidence for convergent emergence hypothesis. Language, genome and protein use an identical layout, which would produce similar profiles even if the models differed. The task difficulty is also not discussed - some tasks are just intrinsically easier to benefit from (near-copies like identity or shifts) and some are hard. Any model that can do simple copying would show a similar profile. The authors do not sufficiently discount a shared task structure, shared prompt formats, non-independent tasks, or noise.

Questions for authors
  1. What is the within-modality reliability of the 100-task profile (e.g., across encoding resamples or model sizes), and how do the cross-modality correlations compare to it?
  2. Please discuss how the approach can be extended to one richer task family (e.g., arithmetic or non-binary alphabets) to test generalization of the profile correlations.
3 comments

We thank the reviewer for recognizing that our paper presents a sound and insightful approach to investigating emergent ICL, has a strong theoretical framework, and honestly presents negative results and caveats. We appreciate your feedback and questions and address them in detail below:

“The task domain is narrow, so generality is limited. All tasks chosen by the authors are 8-bit transformations, many solvable by copying or shifting, so the profiles may reflect simple behaviors rather than ICL in general. The authors could explore either narrowing the claims or add a non-bitstring task family.”

We agree our task domain is narrow, and that our claims should be qualified accordingly. We have added a “Limitations of toy tasks” subsection to our discussion/open-questions section, where we discuss limitations of our results in depth and suggest directions for future work. We also add the subsection “Generality of the cross-modality encoding”, where we describe how our methodology could be applied to non-bitstring tasks.

“The correlation is a weak evidence for convergent emergence hypothesis. Language, genome and protein use an identical layout, which would produce similar profiles even if the models differed. “

We investigated this objection by testing a variety of prompt encoding formats in language and images. We find that across four distinct encodings, the average correlation with the reference encoding is 0.74 for language and 0.82 for image - against ceilings of 0.86 for language and 0.84 for image, computed via correlation of a re-seed reference run. This indicates to us that the tasks a model excels at stay fairly consistent across encodings, even if the magnitude of ICL gain may vary (as some language encodings only offer a few percentage points of gain). This is especially pertinent in language - where the average correlation of the reference run with other modalities is 0.49, which is substantially below the 0.74 correlation with other linguistic encoding types. Thus we think it unlikely that an identical layout is what produced the similar profiles. Additionally notable is that the five-modality block never changed under any linguistic or image encoding - image always remained the outlier. Our full work on this is catalogued in Appendix K.

“The task difficulty is also not discussed - some tasks are just intrinsically easier to benefit from (near-copies like identity or shifts) and some are hard. Any model that can do simple copying would show a similar profile.”

We have added a discussion subsection that addresses task difficulty - “Difficulty scaling” - where we acknowledge that the present methodology does not consider difficulty, and analyze potential future directions that could incorporate it. We emphasize, however, that simple copying via induction head is not enough, as we ensure the query provided is distinct from all in-context inputs - thus there is nothing in-context to start the copy operation from.

“The authors do not sufficiently discount a shared task structure, shared prompt formats, non-independent tasks, or noise.”

We account for noise via bootstrapping over tasks and permutation tests, and report confidence intervals for ICL gain and indicate which correlations are statistically significant. We investigate the impact of shared prompt in Appendix K, and consider shared task structure and non-independent tasks a necessary part of the suite - we make a note to taxonomize tasks in Figure 5, with more detail provided in Appendix J.

“What is the within-modality reliability of the 100-task profile (e.g., across encoding resamples or model sizes), and how do the cross-modality correlations compare to it?”

We already average across specific token assignments in discrete-token modalities by randomizing which tokens are mapped to which bit or separator. At your recommendation, we further investigate the within-modality reliability across encoding types for language and images in Appendix K, as discussed above - we find that the reference language run has an average correlation of 0.49 with other modalities, which is substantially below the 0.74 correlation with other linguistic encoding types. Similarly, for images, we find that the reference image run has an average correlation of 0.17 with other modalities, which is substantially below the 0.82 correlation with other image encoding types. We further investigate correlation across model sizes within a modality, and find that for all five modalities with multiple sizes, within-modality correlation exceeds cross-modality correlation - a full writeup of this work is in Appendix L. We think this provides evidence that the task profile is fairly robust to encoding and model size - indicating that the modality shapes the result more so than the prompt format or model size. We therefore infer that the correlations between distinct modalities are likely not artifacts of chosen encodings.

(cont. in next comment)

(cont.)

“Please discuss how the approach can be extended to one richer task family (e.g., arithmetic or non-binary alphabets) to test generalization of the profile correlations.”

We add a subsection to the discussion section: “Generality of the cross-modality encoding” - where we discuss how our methodology supports any task in which the number of unique elements in the source domain is not larger than the size of the tested modality’s tokenizer. We give concrete examples of where this could apply - such as quaternary arithmetic or latent permutation encoding.

We thank the reviewer for their feedback, especially around the area of designing controls for prompt encodings. The resulting experiments greatly enhance the strength and nuance of our paper. For ease of reviewing, all revisions in our updated draft are highlighted in blue.

Reviewer_R3z9OpenReview ↗

Thanks for the rebuttal

Thanks for the response. The encoding-robustness and model-size analyses largely address my concern about shared prompt layout driving the correlations, though the encoding variants cover only language and images. Task difficulty (including near-copy tasks) and the narrow task domain are handled through discussion only, and are now acknowledged as limitations. Raising my score.

Reviewer_J6RfOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper tests whether few-shot in-context learning emerges across modalities by encoding one suite of 100 bitstring tasks into six model families (language, genome, integer sequences, images, time series, proteins). It finds clean-vs-label-shuffled gains in all six modalities, with per-task effects correlated across five (images being the outlier) and no meaningful effect in chess or music, which the authors take as partial support for their “Convergent Emergence Hypothesis.”

Critical strengths

It's probably good for there to exist a record of few-shot ICL working across a wide range of modalities.

Critical weaknesses

I didn't find a huge payoff to the second research question, "does ICL in different modalities share certain foundational capabilities." It seems to me the authors found that ICL just works across modalities, modulo some mild idiosyncrasies in Section 4.3 that the authors also do not explain.

The one divergence from convergent emergence, that of images, is confounded with how the input modality is represented/tokenized. Out of the 6 modalities the authors examine, 5 are discrete and 1 is continuous, and they find that the continuous modality does not exactly follow emergent convergence. It's also possible the tokenization scheme (downsampling + pixels) isn't optimal for images, or that transformers themselves aren't optimal for images.

If you asked me to guess the answer for the first research question ("does ICL emerge in different modalities") without reading the paper I probably would have also guessed yes.

Questions for authors

Line 75: I think the hypothesis can be even shorter Line 108: "Per-task clean-minus-deranged effects" not immediately clear what this means

I think it's a fairly straightforward paper and it's not immediately clear to me what the right way to improve it should be. Happy to engage with the authors during rebuttals.

2 comments

We thank the reviewer for recognizing that our paper serves as a record of few-shot ICL working across a wide range of modalities. We appreciate your feedback and address your concerns in depth below:

“Line 75: I think the hypothesis can be even shorter Line 108: "Per-task clean-minus-deranged effects" not immediately clear what this means”

We agree - we removed that line entirely, and renamed “clean-minus-deranged effects” to “ICL Gain” throughout the paper.

“I didn't find a huge payoff to the second research question, "does ICL in different modalities share certain foundational capabilities." It seems to me the authors found that ICL just works across modalities, modulo some mild idiosyncrasies in Section 4.3 that the authors also do not explain.”

We further investigate shared foundational capabilities in our revision - we explicitly test whether shared capabilities are robust across encodings in Appendix K, compare capabilities between Qwen and Mistral in Appendix M, and conduct a preliminary investigation into task vectors in Appendix P.

We also extend our “Open Questions” section with several new subsections. Most relevant to your concern is a substantial analysis of relative ICL strength across modalities and the influence of priors (along with Appendix N, which explicitly investigates the influence of bitstring priors in the linguistic domain). Additionally relevant is our added discussion of the platonic representation hypothesis and its link to our task vector investigation mentioned above.

We hope these additions increase the “payoff” of our investigation into the shared foundational capabilities of ICL.

“The one divergence from convergent emergence, that of images, is confounded with how the input modality is represented/tokenized. Out of the 6 modalities the authors examine, 5 are discrete and 1 is continuous, and they find that the continuous modality does not exactly follow emergent convergence. It's also possible the tokenization scheme (downsampling + pixels) isn't optimal for images, or that transformers themselves aren't optimal for images.”

We want to clarify our tokenization procedure here - our image modality is discrete, and takes individual pixels as tokens. ImageGPT is a decoder-only causal transformer pretrained on 32x32 images, with each token corresponding to a pixel - so this matches the context in which ImageGPT was trained. Our continuous modality is in fact TimesFM, which tokenizes patches of continuous values sampled at discrete times - and TimesFM does follow the emergent convergence. Thus we tentatively do not believe discreteness/continuity to be the driving factor behind whether a modality exhibits convergent emergence.

“If you asked me to guess the answer for the first research question ("does ICL emerge in different modalities") without reading the paper I probably would have also guessed yes.”

We emphasize that our contribution goes beyond “does ICL emerge in different modalities?” - a question to which the answer was already non-obvious, as ICL has been studied almost exclusively in linguistic contexts. Our contribution goes beyond this to show that models across modalities can perform ICL on the same shared set of tasks, rather than individual per-modality tasks. We also show that five of the modalities strongly correlate on which tasks they learn via ICL, and show two modalities where ICL does not emerge. Most of this information isn’t specifically predictable from the experimental setup, even if you came in believing ICL emerged in different modalities.

We again thank the reviewer for the detailed feedback, and hope the response above addressed the reviewer’s concerns. All revisions in our updated draft are highlighted in blue.

Thanks for your revisions, which have made the paper clearer. Unfortunately I still don't think I can increase the score on informativeness. I still maintain that the paper is well-written and sound, and that the authors clearly and faithfully represent their results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment by Authors

We deeply thank the reviewers for their time and constructive feedback. Reviewer J6Rf remarked that it was good for there to exist a record of few-shot ICL working across a wide range of modalities. Reviewer R3z9 highlighted our paper's sound approach and strong theoretical contribution, and together with Reviewer swxE praised our paper’s transparency and acknowledgement of negative results. Reviewer swxE additionally described the questions our paper pursued as interesting and our method as novel.

Based on the valuable feedback and suggestions provided by reviewers, we have updated our submission with a variety of additions:

Analysis of Encoding Robustness: We re-ran our language and image results under four alternative encodings, along with a re-seeded reference run, to better quantify the impact of encoding choice on correlation - we found that the primary five-modality block and image’s outlier status are preserved under every single one.

Stability of Task Profiles Across Model Sizes: We analyzed the correlations between the task profiles of models of different sizes that belonged to the same family - in all five modalities with multiple sizes, within-modality correlation exceeded out-of-modality correlation.

A Preliminary Investigation Into Task Vectors: We compare the similarity matrices of task vectors across modalities, and find that four of the five modalities in the five-modality block remain strongly correlated relative to images, with NextTerm as a notable outlier.

A Variety of Further Controls: We test the impact of directly showing Qwen3 bitstrings in a few shot format, and show that this greatly increases performance - which emphasizes the importance of our randomized-digit encoding for fair comparison. We test a second language model family - Mistral 7B v0.3 - and find it exhibits a Qwen3-like profile and correlates with the other four modalities in the five-modality block, but not with images.

An Expanded Discussion Section: We expand our “Open Questions” section into a “Discussion and Open Questions” section, and add paragraphs discussing relative ICL strength and the role of priors, generality of our cross-modality encoding, the limitations of bitstring tasks, the topic of task difficulty, and the Platonic Representation Hypothesis.

Clarity Fixes: We rename “clean-deranged gap” to “ICL Gain”, and semantically reorganize the appendix and add a new appendix table of contents.

We thank all reviewers for their valuable feedback, which greatly enhanced the strength and robustness of our paper and results. All revisions are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.