acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Algorithmic Explanations of Transformer In-Context Learning: A Systematic Study of Noisy Linear Regression

Abstract

Transformer in-context learning (ICL) is increasingly being used to learn statistical procedures, making otherwise costly or impractical inference accessible. Their reuse across datasets makes it important to establish how faithfully they approximate the intended statistical procedure as the inputs change. Nevertheless, constructions showing that Transformers can implement familiar algorithms, together with agreement on selected distributions and isolated mechanistic tests, do not establish that trained models inherit those algorithms’ generalisation properties. We address this gap by systematically studying Transformers trained to perform noisy linear regression in-context. We identify the main features organising their generalisation and how they respond to changes in pretraining, finding substantial competence beyond the training curriculum, but also significant sensitivity to input changes that leave the optimal prediction unchanged. Comparisons with the algorithms proposed in recent literature, supported by theory and targeted interventions, show that the trained Transformers violate structural properties those algorithms satisfy. The resulting characterisation supplies constraints for explanations of statistical ICL and insight for designing models that preserve the properties their intended use requires.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
in-context learning, transformers, algorithmic generalisation, linear regression, meta learning, out-of-distribution generalisation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7iR6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors extend the studies of Naim et al on the limits of transformers and how their performance diverges on regression tasks on inputs from different distribution shifts from what formal algorithms would predict. While Naim et al look at low dimensional linear and polynomial functions, this paper extends the study to large dimensions and defines distribution shifts in terms of spectral diversity (rotation etc). The distribution shifts in particular are dictated around just the choice of input points not on the parameters of the polynomial or linear function and determined by a Gaussian noise. This corresponds to one dimension of the distribution shift in Naim et al.

Critical strengths

This paper advances the study of transformer in context learning for regression problems for multivariate functions. They also offer what to this reviewer are novel bounds on relative risk, which are interesting.

Critical weaknesses

Many of the results echo Naim et al. more than a little bit. For instance at l. 210 the authors find deterioration at the upper end of the pretraining range. This would seem to correspond to Naim et al's boundary values (in KI) but also see arXiv preprint arXiv:2502.03503 for a much more detailed analysis of the causes of failures of generalization. The different training pretraining styles in terms of spectral diversity 238 are similar to the different pretraining regimes countenances in arXiv preprint arXiv:2502.03503. Extending context size in Naim et al's experiments seems very similar to the discussion at the end of 3.4 on extending pretraining counts. And then in section 3.4, unless I misunderstand, the authors find that generalization degrades as the values of the function exceed those in pretraining, also something noted in Naim et al. for both univariate linear functions and polynomials. As the authors acknowledge the layer norm result in 4.2 is also similar to Naim et al.
The bounds given in 4.3 are to new to this reviewer and interesting but it's hard to argue for publishability just on that alone. For this paper to be acceptable, especially in light of the use of gen ai methods on the mathematical formulations, it is important to have an in depth discussion of how the results in this paper go beyond those in Naim et al.

Questions for authors

please motivate the use of noise better. If it's to provide distribution shifts, then the shifts seem very small. What one might call near OOD rather than far OOD. On the other hand, given that many benchmarks are mixtures of quite different problems one would expect that a larger shift would be called for. Can you remark on this?

Please also explain in more detail your experimental setup with covariance

Please describe in depth how you differ from the prior analyses in this area. See papers referenced above.

2 comments

Response [1/2]

Thank you for your detailed review. We agree that Naim et al. are particularly relevant to our work, and that the original introduction did not sufficiently explain the differences between their contributions and ours. We have therefore significantly expanded the introduction, with revisions denoted in red, to clarify these distinctions.

Novelty relative to Naim et al.

We agree that several of our observations have precedents in Naim et al., including failures under changes in input or function magnitude, the effects of widening pretraining distributions, and limitations associated with LayerNorm. These are important contributions which we now discuss explicitly in the introduction.

However, we disagree that our principal contribution is simply to extend their experiments to higher-dimensional regression. Naim et al. principally investigate the limits of recovering underlying functions beyond the pretraining distribution. Their main experiments use noiseless linear and polynomial regression, where sufficiently many observations determine the function exactly. They identify particular extrapolation failures, investigate how these depend on pretraining, and provide explanations for some of the limitations.

Our objective is instead to characterise the approximation to an optimal finite-sample statistical procedure that a trained Transformer has learned. We study noisy regression, where the optimal predictor is ridge, and vary the input covariance while keeping this statistical rule fixed. This allows us to investigate how agreement with ridge depends on the information available in different directions of the regression problem, rather than simply whether a model continues to recover a function outside its pretraining distribution.

Our principal empirical finding is a systematic organisation of generalisation across a rich range of covariance spectra, largely determined by input scale and effective dimension. This organisation remains informative even when we vary the remaining spectral degrees of freedom extensively, and the pronounced dependence on scale persists under substantial changes to pretraining. We investigate both the significant generalisation beyond the two simple spectral families used in pretraining and the structure of the remaining errors. This is a different kind of characterisation from the extrapolation limits studied by Naim et al., and provides a more precise account of the statistical approximation the Transformer has learned.

We also investigate whether the Transformer preserves the mathematical properties of the intended procedure. In particular, Section 3.3 shows that different contexts with identical ridge sufficient statistics can produce substantially different predictions, even after explicitly penalising these discrepancies during pretraining. This also distinguishes our context-count experiment from an ordinary test of generalisation to longer contexts, since we vary the number of examples while keeping the optimal ridge prediction exactly unchanged.

Regarding the large-scale deterioration studied in Section 4.3, it should be noted that we increase the magnitude of the context covariates while keeping the coefficient prior and query distribution fixed. The correct prediction is not required to grow with the context magnitude. Thus, this behaviour is not simply explained by the output-magnitude limitations identified by Naim et al.

More importantly, our systematic characterisation provides a pattern of generalisation against which proposed algorithmic explanations can be tested. A failure to match exact regression does not by itself rule out an approximate iterative algorithm, since such an algorithm may itself fail under distribution shift. In Section 4.3, we show that the finite inverse iteration considered has relative-risk bounds uniform in context scale, whereas a broad class of fixed finite polynomial computations necessarily has diverging relative risk at large scales if it achieves nontrivial accuracy. These results demonstrate how the observed generalisation behaviour can distinguish proposed computational explanations, rather than merely establishing further discrepancies with exact regression.

Response [2/2]

Noise and distribution shifts:

Noise is not introduced to generate distribution shifts. Its purpose is to give a nontrivial finite-sample statistical inference problem, where the optimal prediction involves regularisation according to the information available in the observations. The coefficient prior and noise law remain fixed throughout, and the distribution shifts come from changes in input covariance and other aspects of the context. This allows us to vary the statistical problem without changing the optimal prediction rule.

The shifts are not restricted to small deviations from pretraining. We examine context spectra over several orders of magnitude in scale, with widely varying effective dimensions and spectral shapes. We also deliberately investigate behaviour within and near the pretraining range, since understanding the accuracy and limitations of the learned approximation there is as important as identifying failures under extreme changes.

Covariance experimental setup:

Our reference pretraining curriculum samples covariance matrices whose eigenvalues follow either a smoothly varying log-linear spectrum or a two-level spectrum, with independently randomised eigenvectors. Both families vary broadly in scale and effective dimension. When the number of context examples changes, we rescale the covariance so that the expected context Gram matrix remains fixed.

For evaluation, we instead specify the spectrum of the realised Gram matrix (M=X^\top X), allowing us to vary overall scale, effective dimension and the remaining spectral shape separately. We then sample the remaining context variables and responses according to the Gaussian regression model. This distinction between the pretraining covariance distribution and the realised spectra used for evaluation is important to the systematic characterisation in Section 3.

Reviewer_CqbLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors study the performance of a Transformer performing in-context linear regression under various distribution shifts. The authors show that for various algorithms proposed in the literature to explain ICL in Transformers on linear regression, the behavior of proposed algorithms sometimes deviates from the actual generalization exhibited by their Transformer models.

Critical strengths

The authors explore a timely topic. They present a range of experiments to support their assertions regarding the limited generalization abilities of Transformers on ICL regression, as well as the inability of some proposed algorithms in describing some of these generalization failures.

Critical weaknesses

The novelty of this work appears to be very limited, and many of the author's core claims have already been established in detail by prior work. For instance, the sensitivity of Transformers to distribution shifts when performing ICL on linear regression have been established at length in numerous prior work, starting from the original Garg et al 2023 and particularly Ahuja and Lopez-Paz 2023, among many others. The differences between ICL in Transformers and the algorithms proposed to explain it have also been explored in detail in several prior work including Hill et al 2025 and Anwar et al. While the authors do cite many of these works, the Related Work section of the paper is minimal and does not address the strong overlaps with prior literature. Though the authors do propose some novel distribution shifts and evaluations of proposed algorithms, the general message has already been firmly established in prior work, so the contributions appear to be minimal. It remains unclear what new knowledge has been gained, and how it might benefit future advancement in understanding or engineering.

There are a few issues with the experiments themselves. In particular, it appears that the authors only test a single, specific configuration of Transformer, and it's unclear what the results look like for a sweep over architectures. Experiments also seem to have been conducted with a single seed, giving no indication as to the spread of the results.

Questions for authors
  1. What distinguishes this work from prior literature that already establish the sensitivity of Transformers to distribution shifts, and deviations between Transformer behavior and the behavior of algorithms proposed to explain it?
  2. ICL regression is a very specific setting. How might your results contribute to a broader understanding of Transformers beyond this setting?
1 comment

Thank you for your review. We believe that there has been a misunderstanding as to the scope and novelty of our results compared with the past literature. We have therefore significantly expanded the introduction to make these distinctions clearer, with revisions denoted in red. We have also renamed Section 1.1 Further Related Work to clarify that the main discussion of the relevant literature is contained in the introduction.

We agree that previous work has already established that Transformers can fail to generalise under distribution shifts, and that their predictions can differ from those of classical regression algorithms. However, these observations are not the principal contributions of our paper.

For example, \citet{garg2022} demonstrated both generalisation and failures under selected changes to input distributions, while \citet{ahuja2023} principally investigated only two Gaussian input-mean shifts in experiments. \citet{hill2025} investigated failures under subspace shifts and their relationship to internal representations, while \citet{anwar2025} examined adversarial robustness between Transformers and classical regression algorithms. These are important results, but they do not provide a systematic account of what governs the accuracy of trained Transformers across varying regression problems. Closer to our setting, \citet{zhang2024} investigated how pretraining on varying covariances affects generalisation. Their experiments principally consider random diagonal covariances and a small number of changes in scale. We acknowledge these contributions, and have made the distinctions clearer in the revised introduction.

Our paper instead investigates how accurately a standard deep softmax Transformer approximates the same finite-sample ridge procedure as the covariance structure of its inputs changes. In particular, we identify a remarkably simple two-dimensional organisation of approximation error in terms of input scale and effective dimension, despite the much larger number of degrees of freedom in the covariance spectrum.

Regarding algorithmic explanations, though previous studies have already established discrepancies with classical regression algorithms, a failure to agree with the optimal predictor does not by itself rule out a proposed approximate algorithm, which may itself make errors under distribution shifts. Our systematic characterisation gives a more informative basis for assessing whether proposed computations account for the observed behaviour.

First, we show that the trained Transformer can give substantially different predictions on contexts with identical (M=X^\top X) and (b=X^\top y), despite these being the only context quantities used by ridge and several proposed algorithms, even after explicitly penalising the corresponding symmetry violations during pretraining. Second, we prove that the finite inverse iteration considered in Section 4.3 has relative-risk bounds uniform in context scale, whereas a broad class of fixed finite polynomial computations, including several gradient-based proposals, generally necessarily exhibits diverging relative risk at large scales.

These results show that the algorithms proposed in the literature make different predictions about generalisation, and that the observed failures of trained Transformers cannot simply be attributed to finite computation in general. Our contribution is therefore not another demonstration that proposed algorithms are imperfect explanations, but a systematic characterisation of the learned approximation, together with mathematical tests of whether proposed computations can account for its behaviour.

Regarding the experimental scope, we agree that additional architectures and training seeds would be useful for establishing how widely the observed behaviour holds. Our objective here was to investigate a standard eight-layer, independently parametrised softmax Transformer under controlled changes to its pretraining and evaluation distributions. This limits the generality of the empirical conclusions, but does not invalidate the reported behaviour of the trained models or the mathematical results concerning the proposed algorithms, and so we believe it is sufficient for the principal conclusions drawn.

Finally, regarding broader significance, we do not claim that our findings directly describe off-the-shelf LLMs. Rather, statistical ICL provides a setting in which we know the intended inference procedure exactly and can investigate what the Transformer has learned relative to it. The identification of statistical quantities organising generalisation, together with tests of whether the learned model preserves the properties of the intended procedure, gives a more precise basis for developing and evaluating algorithmic explanations. This is particularly relevant to learned statistical inference methods such as PFNs, where the objective is to approximate an inference procedure that can be reused across datasets.

Reviewer_cEPkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate when transformer ICL agrees or diverges from classic statistical methods such as ridge regression. This builds on observations from the literature showing limitation of ICL generalization to scaled-up inputs or distribution shifts, even when transformers have been trained from scratch on synthetic data. The authors tackle this question by analyzing how ICL behavior responds to changes in the pre-training distribution, focusing in particular on architectural components like layer norm.

Critical strengths

I share the authors perspective that the mechanistic explanations for ICL have in some sense been overstated in the early work, and so we need good characterizations of how ICL depends on pre-training data.

Critical weaknesses

I find the paper very hard to follow, and ultimately was unconvinced that their results have meaningful implications for practical ICL settings (e.g. off the shelf LLMs). I suspect heavy LLM usage has negative affected clarity. For example, sentence construction is occasionally very strange (use of passive voice, run-on sentences, etc.) in a way that makes it hard to follow [e.g. L025-027, 092-094]. The presentation of the theoretical results was quite dense. The heavy use of in-line numbers and perecentages is another telltale sign of heavy LLM use in the writing. The empirical setting does not seem connected to what practitioners care about, and there are missing details in terms of basic explanation of what is evaluated (e.g. GD++ is not described formally).

Questions for authors
  • why Eqn R [L406]?
  • why focus on per spectrum relative risk [L197]?
  • what is the relevance of your results to practical ICL settings?
1 comment

Thank you for your review. We believe that most of the criticism reflects misunderstandings about the paper’s scope and contributions. To this end, we have revised the introduction to make these clearer, with new material marked in red. While the review expresses significant concerns about soundness, it does not identify a particular incorrect theorem, proof step or experimental result. We therefore focus on addressing the specific technical questions raised above.

We regret that the manuscript was found difficult to follow and accept full responsibility for overusing the passive voice and for the dense presentation of in-line numerical results. Yet, we would caution against attributing every instance of awkward academic prose to an undeserving LLM: some of us are perfectly capable of producing it unaided.

More substantively, this paper was not aimed at directly addressing ICL in off-the-shelf LLMs. Even in as simple a setting as multi-layer Transformer regression, the statistical procedures implemented by the Transformer, and even the extent of its accuracy is poorly understood. There has been a significant recent literature attempting to contribute to this understanding, and competing approaches to whether simplification for mathematical tractability can yield understanding applicable to a full multi-layer Transformer.

In this context, requiring direct applicability to off-the-shelf LLMs would be to impose a different objective on the paper. Our aim is instead to understand and characterise the generalisation of trained Transformers in a setting where the optimal statistical procedure is known. In particular, we show that agreement with ridge across a rich range of covariance spectra admits a largely two-dimensional organisation in terms of scale and effective dimension, with this structure persisting under significant changes to the pretraining distribution. This provides a concrete pattern of behaviour that proposed algorithmic explanations should account for, and our theoretical results in Section 4 begin to assess them against it. The relevance to more practically important settings, such as PFNs, is in providing a way to study how faithfully a learned inference procedure approximates its intended statistical target across different datasets, rather than claiming that our particular regression findings directly extend to those models.

Regarding the more isolated specific comments:

GD++: While Section 4.1 describes the central mathematical mechanism of GD++, including the transformation of the covariates at each iteration, we do not give its full recurrence. The properties relevant to our results are its dependence on the context through the Gram matrix and response statistic, and its finite polynomial computation structure. A complete description of the algorithm was not needed for these arguments and would have introduced additional detail without changing the analysis.

Equation (R): We define radial affinity to encompass the query dependence of all the algorithmic proposals considered. Most predict linearly in the query, but the single-layer ReLU-attention predictor analysed by \citet{ahn2023} need not be query-linear while remaining affine in query magnitude along each fixed direction. Condition (R) includes this case.

Per-spectrum relative risk: We use relative risk because it measures the error in approximating ridge relative to the magnitude of the optimal prediction. Without this normalisation, differences in error can arise simply because different spectra produce very different amounts of predictable signal, rather than because the Transformer approximates the statistical procedure more or less accurately. In particular, the relative risk is zero for ridge and one for the zero predictor. We calculate this separately for each spectrum and take the median to characterise typical performance, rather than weighting spectra according to the magnitude of their predictions. We also considered other quintiles and search over spectra at fixed scale and effective dimension to characterise the spread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.