acceptodds
Under review as a conference paper at ICLR 2027

You Don't Need Feature Learning to Do In-Context Learning

Abstract

How should we understand memorization vs. generalization in in-context learning (ICL) tasks, and when does one happen vs. the other? To investigate, we study how rotation-invariant kernel machines learn the ICL linear regression task proposed by Raventos et al. (2023). By decomposing the task in the kernel eigenbasis, we obtain a spectral picture of learning in this ICL setting. This picture reveals that as sample size grows, kernel machines learn successively-higher-degree components of the memorizing solution, which depend upon successively-higher-degree statistics of the training task set. The generalizing solution is approximately a low-degree truncation of the memorizing solution, and so the model first generalizes, then degrades into memorization as training proceeds. The takeaways are twofold. First, if this case is indicative, learning a “generalizing” solution on an ICL task is really just underfitting the memorizing solution! Second, the fact that kernel machines can generalize on this task means that feature learning is inessential to ICL.

Primary area:
learning theory
Keywords:
in context learning, kernel theory, learning curves, feature learning, overfitting, spectral theory, first principles

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dnh3OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes an explanation the generalization versus memorization phenomenon for the linear in-context learning setting as a demonstration of spectral bias—first learning the low-degree components, which allows better generalization, before learning the high-degree parts, leading to close memorization of the training data but worse performance on novel tasks. The work quantifies this behavior by performing a novel eigendecomposition of the closed-form solutions of empirical and population loss for linear ICL, then observing how these components are learned by a kernel machine.

Critical strengths

The perspective is interesting and natural, extending a well-observed framework to understand a relatively new phenomenon and give it a mathematical grounding. The idea is well-motivated.

The subkernel expansion framework and application for an approximate kernel decomposition in this setting is creative and promising, even if its correctness is only empirically observed here.

Critical weaknesses

The evidence, both theoretically and empirically, for an actual spectral bias in this setting is not very convincing. The experiments focus on one case (input and task distribution) and the confirmation is based solely on visual plots.

Some of the conclusions made/suggested seem exaggerated and overblown, especially given there is not an analysis of training dynamics, extensions beyond linear models, or architectures used in practice for ICL (i.e. transformers).

Questions for authors
  1. Please clarify the relationship between TT and nn. To me, it reads that TT is the number of training tasks but the model is trained on nn contexts (tasks) drawn from those TT, then you take nn to infinity… so you are reusing samples? What is nn in Figure 3?

  2. You include noise in your model: what is the effect of noise on generalization/memorization? Does the noise act as implicit regularization, like in standard settings? What about noisy observations of the tasks (coefficient vectors)?

  3. In line 283 you mention the study is in the L≫dL\gg d regime—is this mentioned earlier? If not, it should be.

  4. In Figure 2, the increase in novel task MSE after learning generalizing solution does not look significant, at least visually. Also, this seems different from the usual spectral bias, which appears over optimization steps. Here you train with online SGD (so I guess it technically matches), but in practice training is done on the whole training dateset (or as batches), so I thought you would plot number of steps/epochs—can you elaborate on this difference and why you use online SGD? How is this different from grokking?

  5. Any explanation of why the training dynamics would result in spectral bias in this setting? Beyond the empirical results. Is it just like NTK?

  6. You take kernel regression with a particular kernel as the learning algorithm, and all analysis is based off of this. Can this algorithm be achieved by actual architectures (i.e. ICL with transformers) and would this same kind of phenomenon necessarily hold? If that isn't obvious, I think it’s a very big jump to suggest this phenomenon generalizes beyond linear tasks and with real-world models.

  7. One of the conclusions you make (which is also the title) is that feature learning is not necessary, which is technically true since you use fixed/random features, but it doesn’t seem like that should be the main takeaway. Isn’t this known? Transformers can implement in-context closed form estimators, implicit GD, etc.

  8. What does this provide over usual generalization error bounds with bias-variance tradeoffs, which exist for ICL?

  9. Typo on line 95: “region.gm”

4 comments
AuthorsOpenReview ↗

Response to Reviewer Dnh3 (Weaknesses section)

We thank the reviewer for the feedback on our paper. We appreciate that the reviewer found our perspective interesting and natural, in addition to finding our subkernel expansion framework creative. Here, we address the reviewer’s major weaknesses:

Evidence for spectral bias: This seems like a non-issue to us: kernel regression is already well-understood to obey a spectral bias, as has been thoroughly demonstrated theoretically and empirically (see e.g. [https://arxiv.org/abs/cond-mat/0106475, https://arxiv.org/abs/2002.02561, https://arxiv.org/abs/2006.09796, https://arxiv.org/abs/2110.03922]). It is not necessary for the present work to show that kernel regression learns with a spectral bias given by the kernel eigensystem, as this is already known to hold generally.

Evidence for the provided story: Indeed, the strongest evidence that we have correctly computed the kernel eigensystem is the fact that experiment dots fall on theory curves. This is strong evidence (e.g. note the clear match in Figures 1, 8, 9) and is standard. However, we also provide non-visual evidence for our story in Section 3 which discusses decomposing the memorizing and generalizing solutions into the kernels eigenbasis, finding that the two are approximately equivalent when the solutions are truncated at degree-3 polynomials.

No training dynamics: As with spectral bias, the training dynamics of kernel regression follow trivially from its eigensystem. We also note that we do study the training dynamics of RF regression trained by SGD in Figures 2 and 3.

Applicability to non-kernel settings: We refer the reviewer to our general response concerning transformer models.

AuthorsOpenReview ↗

Response to Reviewer Dnh3 (Questions section, part 1)

To the reviewer’s questions:

1. Task diversity versus sample count: The task diversity TT is fixed from the outset of the problem, we then train models using nn samples taken from the distribution with TT tasks. For kernel regression models, this number of samples nn is fixed, although our theory assumes n→∞n \to \infty, as is customary in kernel theory [see e.g. https://arxiv.org/abs/2006.13198, https://arxiv.org/abs/2510.14878]. For Figure 3 (Figure 4 in the updated pdf) which shows the same memorization to generalization transition as is in Raventos et al., we train models on distributions with various task diversity on 4⋅1064\cdot 10^6 samples, plotting what their error is at the end of training.

2. Noise: There are various aspects of noise that appear in our model, and one that affects the data itself. There is the noise in the ICL problem itself, which is added to the in-context labels themselves (and sets the scale where if the training tasks have a covariance below the noise threshold, these directions are poorly resolved by model, described in Appendix H.2, line ~1680). Our general finding of the generalization-memorization transition is, however, largely independent of noise effects. We have updated the paper, adding in a sentence in line ~424 to clarify this point for future readers. There is also noise in the eigenfunctions themselves (being a data-dependent observable) along with their fit coefficients, however this noise is averaged over by taking multiple trials. We leave a discussion of this noise to the kernel learning curve papers, such as Simon et al. 2021: https://arxiv.org/abs/2110.03922.

3. Context length regime: We work in a regime of only L≥dL \ge d, although we agree that this was not well articulated in the submitted version of the paper. In our updated version, we make this more transparent, and have added in a section (Section 6.1) that describes the effect finite context length LL has on the generalizing, memorizing, and kernel solutions.

4. Significance of the rise of the error, SGD: We agree with the reviewer that it is difficult to visually see a dramatic rise in the test error on novel tasks, but note nonetheless that a rise in the error is indeed visible. We are actively working on finding a setting where the rise is more stark. As for the spectral bias, we do indeed use online SGD such that each update uses a fresh sample, with the number of optimization steps and number of seen samples coinciding (up to the batch size of 1024). We chose to use the number of seen samples instead of the number of optimization steps to better align with the kernel results where the solution is found analytically.

4.1 Grokking: Grokking is a separate phenomenon entirely, where the generalizing solution is capable of interpolating the entire training dataset (contrast this against the ICL setting wherein the generalizing solution is always suboptimal and cannot interpolate the training dataset). In grokking, should a model learn the memorizing solution, then it will never generalize without external pressure (i.e. weight decay), whereas in ICL, a model will generalize first then memorize as a byproduct of trying to achieve the lowest possible training error.

AuthorsOpenReview ↗

Response to Reviewer Dnh3 (Questions section, part 2)

5. Spectral bias: Please see the above point of “evidence for spectral bias.” Yes, for linearized models, the model is well-described by the kernel’s neural tangent kernel (see Jacot et al. 2018: https://arxiv.org/abs/1806.07572).

6. The learning algorithm: We chose kernel regression for the above reason such that we could describe the inductive bias, and thus the error decomposition, of the networks. Practically speaking, any model (including transformers) that is a universal approximator can achieve the results we detail, although we do not have as well developed theories of other model’s inductive biases. Please refer to the general response concerning transformers. On “generalizing beyond linear tasks,” a kernel method is linearized with respect to its weights, not the data, so in theory one could perform a similar kernel analysis on more complicated tasks.

7. Isn’t this known: No. The result that kernel machines are able to perform ICL is not known in the literature. The reviewer is correct in pointing out that feature-learning transformer models can implement e.g. linearized models in-context, however this does not imply that a fixed-feature model will implement such linearized models in-context.

8. Comparison to bounds: The main scientific message we wanted to convey in this paper is the mechanism behind why ICL happens, and the phases of learning, something a bound is unable to express. Additionally, most theoretical accounts of ICL (including e.g. https://arxiv.org/abs/2310.08391 which we believe is the reference the reviewer was referring to) consider only one-layer linear attention models, which are unable to memorize the training tasks and are therefore unable to capture the generalization-memorization transition key to the ICL setup. An error bound will also be unable to capture dip-then-rise behavior in test error throughout time, and would thus not add much to the scientific insight.

9. Typo on line 95: Fixed; we thank the reviewer for pointing this out.

Thanks for the detailed response and updates to the paper; they helped clarify a lot. The transformer experiment is a welcome addition. A more mechanistic study of to see whether those higher degree components emerge during training as predicted would be nice, but even just tracking MSE as is done here gives me more confidence that the theory can offer insight into transformer training.

The spectral perspective is promising but I think the paper as a whole can still be substantially strengthened, including the exposition. Nevertheless, I will increase my score to a 3.

Reviewer_we2UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors study memorization and generalization phenomena in In Context Learning using kernel machines trained on a linear regression task. The authors argue that the model learns the generalizing solution first which degrades into a memorizing solution as training continues. They further claim that since ICL is observed in kernel machines, feature learning is not essential for ICL.

Critical strengths

Providing an analytic understanding of ICL is important Results seem intuitive and are communicated well through the figures

Critical weaknesses

It’s unclear whether any of these insights generalize to more complicated/realistic settings where ICL is studied more commonly

You make the strong claim that your results give the first end-to-end theory of the generalization-memorization transition in ICL. In what sense is https://arxiv.org/pdf/2405.11751 not an end-to-end theory?

Also, can you show that your theory recapitulates the Figure 4 rightmost panel from Raventos et. al? This seems necessary for an end-to-end theory

Questions for authors

It is not clear to me how much of the phenomena the authors describe is due to studying kernel regression instead of more complex models such as transformers. Due to the convexity of kernel regression, the model cannot get “stuck” in one minimum, and so it is not clear that the transience results described here will generalize to the non-convex setting (which transformers fall into). Could the authors clarify this point?

In other words, it is not clear whether transience in kernel regression is a distinct phenomenon to transience of ICL in transformers, could the authors try to link the two more explicitly?

Could the authors expand the discussion section to connect back to the broader picture?

4 comments
AuthorsOpenReview ↗

Response to Reviewer we2U

We thank the reviewer for the feedback on our paper. We are glad that the reviewer found the results intuitive and well-communicated. Here, we address the reviewer’s major weaknesses:

Generalizing to more realistic settings: In regards to transformers, please refer to the general response. For other ICL tasks (see e.g. the ICL Markov model task in "Competition dynamics shape algorithmic phases of in-context learning” by Park et al., https://arxiv.org/abs/2412.01003), the same generalization-memorization transition occurs, and there is no strong evidence in the literature for why the mechanism in those other settings would be distinct from the one of ICL linear regression we have characterized.

End-to-end theory: The linked paper is indeed an end-to-end theory, but it is not one of the generalization-memorization transition in ICL. That is, there is nowhere in the linked paper where the reviewer will find the test error on novel tasks rising while the test error on the training tasks decreases. This is by construction of their theory: a one-layer linear attention layer simply cannot express the memorizing solution. This is in contrast to our kernel models, which are universal approximators. We have updated the related works section to highlight the difference between our theories.

Rightmost panel of Raventos et al. Fig. 4: No, this model cannot capture a sharp transition from generalization to memorization. Such a transition would require a comprehensive theory of feature-learning networks, which does not currently exist. We can use our theory to hypothesize why such behavior occurs (namely the training task distribution being nearly exactly Gaussian), but nothing beyond that. That said, we are unsure why the reviewer sees this as necessary for an end-to-end theory; this is just one phenomenon that occurs in transformers but not in KRR, and our account of KRR on this task is still complete and self-contained.

To the reviewer’s questions:

Convexity of kernel methods: Our methodology was to remove as many confounding effects as possible, with non-convexity of the loss landscape being one such effect. Despite having removed the possibility of the model to get “stuck” in a saddle-point or local minimum, we still observe the exact same generalization-memorization transition that exists in transformer model training. We believe it would be highly unlikely to attribute generalization to non-convex training when the exact behavior exists when training a model with a convex landscape.

Transience in transformers: Please see our general statement of transformers.

Broader picture discussion: Yes, we have updated the discussion section ending our paper to connect back to the broader picture.

Thanks for the helpful response! This clears up some of my questions/concerns, but I'd like to press a bit on a few things:

  1. End-to-end theory: After reading your response, I think we have a nomenclature issue here. You say the linked theory is not a theory of the "generalization-memorization transition in ICL" but later, in response to the "rightmost panel" question, you say about your model "No, this model cannot capture a sharp transition from generalization to memorization."

The language used in the paper, and even in this rebuttal response, currently mixes the training-time transition (where the transition is driven by the number of steps) with the data-driven transition (driven by the number of tasks in the pretraining distribution). It would be great if this could be clarified in the paper, the paper I linked gives, in my opinion, end-to-end theory for the second of these transitions.

  1. Convexity: please see https://arxiv.org/abs/2412.00104 for a theory of a landscape-driven training-time transition. I think it is important to disambiguate between the two distinct mechanisms, and investigate which actually happens in transformers. I realize that your paper is primarily theoretical, but connection must be made with the phenomena being studied. "All models are wrong, but some are useful".

(I also don't quite see why https://arxiv.org/abs/2412.00104 would not constitute an earlier "end-to-end" theory of the transition studied in the authors' paper, I still think this language is far too strong)

AuthorsOpenReview ↗

Response to comment by Reviewer we2U

Thanks for responding so quickly! We're happy to discuss things further.

  1. For the end-to-end theory, we still are unsure why our paper would not consitiute an end-to-end theory of the memorization-generalization transition. The theory in the linked paper cannot memorize as the model studied is linear attention, and does not have the expressivity to memorize tasks.

1.1. As for the precise wording of our rebuttal to Raventos et al. Fig. 4's rightmost panel, we want to be more specific what we mean by having a "sharp" transition from generalization to memorization. The Figure 4's panel itself does not show that the transformer goes from a generalizing solution to a perfectly memorizing solution in a sharp manner (which we indeed cannot describe), it only shows the task diversity at which their model stopped overfitting/memorizing the training tasks. For more evidence that even transformers aren't sharp in their transition from generalizing to memorizing, we refer to Figure 2 of Raventos et al. where a transformer with ~282^8 tasks has error that is notably between the two oracle solutions (something we see as well in our own kernel experiments, where we refer to Figure 4). The timescale at which the memorizing terms are not learned is a prediction that can be made using our theory, however it would be wholy unuseful if applied to a feature-learning model's dynamics (where even a linear target has a clear deliniation in learning speed between feature-learning and lazy models). For these reasons, we don't believe explaining Figure 4 is necessary to have an end-to-end theory of the generalization-memorization transition.

On the training-time vs data-driven transitions, we apologize for any confusing language on our part. The two are intertwined, with our theory making this connection quantitative. As the model learns higher order terms (phase 3 of training), then the model incorporates higher-order statistics of the task distribution. On a separate axis, if we draw more tasks, the task distribution itself becomes more Gaussian, lowering the relative sizes of the higher-order statistics (hence affecting the amount of memorization possible during phase 3). We included Section 6.3 to make this point, although we did not specifically highlight this subsection in our earlier response.

  1. On convexity, we would like to highlight that the generalization to memorization phenomenology of https://arxiv.org/abs/2412.00104 is purely a byproduct of having regularization on the weights (where the ICL-based attention weights are penalized more heavily than the "IWL"-based MLP weights). This account actually goes against the original paper describing transience (https://arxiv.org/abs/2311.08360) which found that weight regularization promotes ICL. In the problem setup of https://arxiv.org/abs/2412.00104, the optimal ICL solution also interpolates the training distribution, which is not generally true. There are a host of differences between our two setups which we feel makes any direct comparisons difficult to make. From this perspective, we don't believe there's compelling evidence that effects specific to non-convex loss-landscapes will explain transience.

On why https://arxiv.org/abs/2412.00104 doesn't constitute as an end-to-end theory: in addition to the reason given above about weight regularization, their paper does not overlay predicted curves of their theoretical model on experimental data. They fit slopes (see e.g. the critical exponent of ν≈0.7\nu \approx 0.7) and measure them to get their theory working, which doesn't constitute as end-to-end.

Thanks for the informative response! While I still have some small concerns, I believe the authors have addressed enough for me to raise my score to a 3.

Reviewer_UqQtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides a spectral theory perspective for interpreting in-context learning. Theoretical analysis is based on simple linear function learning and assumes a kernel regression framework without sequential structure. Authors claim that ICL transiense can be explained the kernel machine learning gradually higher terms: from mean value, to phase 2 with degree 3 truncation, and finally to overfitting data. Their experiments on the similar setup concurs with theoretical claims. The whole setup seems independent from the sequential structure in input.

Critical strengths
  1. provides the first end-to-end theoretical account of the generalization-to-memorization transition in ICL, claiming that generalization is a low-degree polynomial truncation of the memorizing solution
  2. The theory successfully predicts behavior of experiments (but is expectable with the relatively clean experiment setup)
Critical weaknesses
  1. The paper setup is regardless of the sequential structure of inputs, and actually applicable to any machine learning problem or setup. The merit of the paper, therefore, lies in predicting the generalization and memorization progress in a general machine learning setting, but the relation to in-context learning is weak. It fails to predict the unique surprises that in-context learning has provided, where this behavior emerges in a general natural language modeling (on sequential input) process.
  2. The analysis is restricted to the ICL linear regression, with input length of 1. However, I admit that most other papers, except for one or two not cited in this paper, also use these settings. Considering this, I say the importance of this weakness can be moderately lowered.
  3. The experimental validation uses relatively small-scale settings. The model chosen is also vastly different (and simpler) from the setup of in-context learning in most other practical or research settings.
  4. kernel regime corresponds to a infinite-width limit of neural networks, limiting the generality of the discussion.
Questions for authors
  1. Can the discussion be generalized to non-linear classes?
1 comment
AuthorsOpenReview ↗

Response to Reviewer UqQt

We thank the reviewer for the feedback on our paper. We are glad that the reviewer saw the novelty in predicting the generalizing-to-memorizing behavior of ICL from first principles. Here, we address the reviewer’s major comments and questions:

  1. Sequential structure of the data: While we agree that the kernel models we study are unable to dissect the sequential structure of natural language data, it is incorrect to assert that sequential structure is essential to ICL. The key aspect of ICL which we describe in our setting is instead the fact that training samples are generated from a finite training set of tasks, and the learner must generalize to unseen tasks. We thus strongly disagree that the paper’s merits lie in predicting generalization and memorization in “a general ML setting.” Before our paper, there did not exist a first-principles theory describing precisely why models would learn the generalizing solution, or why this solution would change to become the memorizing solution. We ask the reviewer to elaborate what the “unique surprises” of ICL are, if they are not the paradoxical finding that a model would generalize to novel tasks instead of memorizing the training tasks.
  2. ICL Linear Regression setting: We’re unsure what the reviewer means when they mention input length of 1–our contexts have a general context length LL, usually around 1010.
  3. Small-scale setting: We agree with the reviewer that the chosen model is indeed vastly simpler than the setup of other in-context learning analyses. In fact, this is one of the main points of our paper–that ICL is not some nebulous phenomenon that can only be studied in highly complicated models, but it can be studied even in a very simple setting. While the reviewer sees this as a negative, we believe this is a highlight for theoretical pursuits.
  4. Kernel regime: Similar to the previous point, we view this as a highlight of our theory. Our current theories of feature-learning networks are often unspecific to the problem (e.g. DMFT in “Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks” by Bordelon and Pehlevan, https://arxiv.org/abs/2205.09653), and so any theoretical account of ICL will require some model simplification.

Question re: nonlinear classes: Yes, hypothetically speaking the theory can be extended to nonlinear families of ICL problems (thanks to the subkernel expansion framework for finding the eigenfunctions being dataset-agnostic). We don’t perform said analysis as even this ICL linear regression has proven to be highly complex, should one attempt to analyze it theoretically.

Overall comment on weaknesses: To address a main theme throughout the four weaknesses posed, we would like to emphasize that our paper is theoretical in nature, and is therefore listed under the “learning theory” primary area. It appears that most of the reviewers' concerns are linked to more empirical pursuits–namely that we do not predict behavior in the general natural language setting and the model is chosen to be simpler than those used in practice. The goal of our paper is indeed to be simpler than the general setting with powerful models; even in this simple regime we study, there has yet to be first-principles, foundational work that may eventually allow for the more general setting to be studied.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response concerning theoretical extensions into transformer models

We thank the three reviewers for their time reviewing our paper. A common thread in all reviews was a request that we use our provided theory to make a statement about how transformers perform and acquire ICL capabilities in some capacity. While we want to express that our paper is primarily theoretical and constructed around the surprising fact that models unable to do feature learning are able to do ICL, it is clear that including experiments testing our theory in transformer models would make for a more compelling paper. To this account, we have included a new section at the end of the updated version detailing a transformer experiment that tests our theory. In brief, we show that task diversity is not the fundamental quantity that causes ICL, even in transformer models, and instead, the model's low-degree truncation of the memorizing solution is what drives ICL performance. More specifically, we consider two distributions of training tasks with the same task diversity, and show that when we zero the training task distribution’s skewness and kurtosis (which are related to the optimal coefficients of the phase-3 memorizing terms--those of degree 5 and 7 eigenfunctions) that a GPT2-scale transformer trained using the Adam optimizer will generalize on novel tasks for substantially longer than random i.i.d. sampling, as is predicted by our kernel-based theory. This is a novel finding about the learning dynamics of transformer models that is not present elsewhere in the literature, and it was obtained by analyzing the solutions of kernel models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.