acceptodds
Under review as a conference paper at ICLR 2027

When Can Behavioral Tests Distinguish In-Context Computations?

Abstract

In-context learning in transformers is often studied by comparing predictions with familiar algorithms. We ask when a benchmark can distinguish these computa- tional explanations. For linear regression, our test compares a model with the best possible output of a specified family of gradient-style updates, allowing all scalar settings to be chosen separately for each prompt. Lower model error excludes the whole family, rather than one optimizer configuration. On a released 12-layer transformer and one independently trained replication, shorter contexts make pre- dictions less accurate but provide stronger risk-based exclusions. With 22 ex- amples in 20 dimensions, both models outperform the best 12-update predictors, including in an exploratory analysis removing the hardest tenth of prompts. With forty examples, the same families are already more accurate than the models. An exact Gaussian residual law explains the comparator’s improvement: longer con- texts lower its minimum achievable error. A separate symmetry analysis distin- guishes useful departures from errors that depend on example mixing and feature coordinates; at forty examples, these dependencies account for 96–97% of error in the recovered regression coefficients. Prediction accuracy and the strength of behavioral evidence are different objectives: an easier prediction task can make a computational explanation harder to exclude.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, transformers, linear regression, gradient descent, Krylov subspace methods, behavioral evaluation, interpretability, symmetry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YdynOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors appear to study whether Transformers implement gradient descent when performing in-context linear regression. To do so, they compare the loss attained by the Transformer with the loss attained by gradient descent after a variable number of steps. They suggest that for small sample sizes, Transformers can readily exceed the performance of gradient descent trained on a number of steps equal to the model depth, suggesting that the learned algorithm is different. However, their comparison is inconclusive for larger sample sizes when gradient descent attains a lower loss.

Critical strengths

The authors study an interesting topic, and the general direction may hold promise.

Critical weaknesses

I found the manuscript to be excessively difficult to read. The organization of the text is very unusual, and ideas seem to be introduced and referenced out-of-order. The authors also appear to use unusual jargon that isn't defined or motivated. Formal results appear to be injected into the manuscript without much introduction or motivation, further making the manuscript difficult to parse. From the AI use statement, it sounds like the manuscript was written largely or entirely by LLMs, and the haphazard style does seem plausibly consistent with this. The manuscript would benefit substantially from a large editorial pass to ensure the writing and message is coherent.

The central message, as far as I was able to understand, was that Transformers don't seem to be implementing exact gradient descent when trained on in-context linear regression. Their method is sensitive to this conclusion for small in-context sample sizes, but fails to make this distinction for large sample sizes. While an interesting result, it's novelty is unclear, as this conclusion has been exhaustively studied and well established since von Oswald 2023. It remains unclear what additional knowledge was produced.

Questions for authors

None

1 comment

Thank you for connecting the paper to von Oswald et al. (2023). We agree that departures from ordinary GD, including the GD++ behavior they studied, are not new. Our contribution is a quantitative test of whether any member of a specified family can approximate the transformer’s predictions.

A GD run returns one coefficient vector. Our comparator instead selects the best vector from the entire span allowed by the budget, using the true regression coefficient to make that choice (Eq. 2; Proposition 1). A particular GD run can lose because it was poorly tuned; the transformer’s advantage over this oracle cannot be explained that way. At 22 examples both 12-layer models beat even the budget-16 unpreconditioned oracle (Table 1).

The approximation result goes further. On each model’s reported simultaneous confidence event, every budget-12 unpreconditioned predictor is farther from that model’s predictions than the true target is (Table 11). We agree that the presentation made the central contribution hard to follow. We should have made the distinction between a single GD run and the family-wide comparison clear from the start. The guarantee is about how closely an entire tested family can match the observed predictions; it does not identify the model’s internal optimizer or rule out GD++ in general.

We ask that you reconsider the soundness and overall ratings in light of the family-wide guarantee in Proposition 1 and the approximation-distance bound in Table 11.

Reviewer_EZhVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

In recent years, many researchers have attempted to identify the algorithm that a transformer learns to perform in-context when trained on examples of linear regression. This process typically involves comparing the predictive loss of the trained transformer with the loss of a specific instance of a candidate algorithm. The authors of this work suggest that comparing with one deployed instance of a learning rule may be too pessimistic, and instead propose comparing performance against an entire family of predictors from that learning rule with each predictor selected for a specific prompt/test example. This provides a theoretical floor for the loss achievable by predictors within that family, and so a transformer achieving loss less than that floor should disqualify that algorithm family as a potential explanation for ICL in transformers. Across a thorough set of experiments, the authors demonstrate how their technique can exclude a family of predictors based on gradient descent and preconditioned gradient descent as potential algorithmic explanations.

Critical strengths
  • Understanding in-context learning is an important topic. The authors contribution of providing a test to exclude a family of algorithmic explanations is novel.
Critical weaknesses
  • [W1] The paper is very difficult to understand. Paper specific terms are quickly introduced without the necessary context. For instance, in the paragraph on line 37, there is mention of “update budgets” and “combination(s) of update directions” despite these concepts not being defined or clear from the context of the introduction. Even throughout the work, certain terms are never defined, such as “audits”, “Readouts”, “episodes”, and “teachers” in section 4. These terms have a wide variety of meanings in the literature and so an exact definition is necessary for understanding their significance with respect to this paper. The phrasing is oftentimes confusing, such as “averaged unaveraged Haar grid cells” in section 5.4, and referring to a “right hand side” on line 111. The motivations for several aspects of the work are not clear. For instance, why is it important to consider the span of gradient descent updates instead of a single solution (see Q1)? All of this contributes to the work being difficult to grasp.
  • [W2] It’s not clear what the impact/implications of this work are. Some of the theoretical results are unsurprising and therefore not insightful. For instance, it is expected that expanding the context would also improve the performance of the “rival” predictor class that we are comparing against due to typical variance reduction arguments. More importantly, it is not clear how to interpret the exclusion of related predictor families, or even the same predictor family at a smaller number of examples. For instance, in figure 2 we see that 12 steps of gradient descent is universally excluded at 20 examples, but is not excluded at 40 examples. Does that mean that we can’t rule out 12 step gradient descent as an explanation for the transformer’s performance on 40 examples, but we can for its performance on 20 examples? In both cases, I am unsure about what conclusions could be drawn from the results presented in the paper.
Questions for authors
  • [Q1] Is it not a meaningful limitation for the gradient based optimization procedures to effectively have to commit to a certain number of steps? Is this to adjust to adaptable learning rates? Why is it important to consider the span?
  • [Q2] How should we interpret the results if we can exclude gradient descent with some number of steps on some number of examples? Does this give us any definitive idea as to whether these transformers implement gradient descent?
1 comment

Thank you for the candid feedback on the presentation. We agree that the paper was hard to follow: we introduced too much terminology before explaining what the test measures and how to interpret it. A single GD run can lose because its learning rate or stopping point was poorly chosen. The span gives the comparator all those choices. The oracle takes the best coefficient vector in the span for each prompt, using the true coefficient to choose it. The budget limits the available directions, not the choice of scalar weights or stopping point. Beating this optimum rules out the specified family rather than just one schedule.

On the 20-versus-40 comparison: yes, the short-context risk test rules out the specified budget-12 family, while at 40 examples the risk test alone is inconclusive. A predictor can match the model’s error without matching its predictions (Appendix B.9). A separate test still rules out the budget-12 unpreconditioned class at 40 examples: joint averaging over row mixing and feature rotation preserves the class (Proposition 5), but both averaged predictors retain useful information outside its span (Table 12). If the original predictors belonged to the class, the latter quantity would be zero. Neither test identifies the optimizer the model actually uses.

More examples can improve the comparator even here, where the noiseless support already determines the true coefficient. The remaining error comes from the restriction on the comparator’s directions. Proposition 3 characterizes the Gaussian floor, and Corollary 4 gives a condition under which model error falls while the risk comparison loses its ability to exclude the family.

For clarity, in our experiments, we use “episode” to refer to one sampled regression task and “teacher” to refer to the coefficient that generates its labels. We use “readout” for the map from a hidden state to a prediction, and “audit” in Section 5.4 for an evaluation on transformed versions of a task. The distinction in that section is about the order of operations: we can measure each transformed prediction and then average the measurements, or average the predictions first and then measure the result.

In light of what the two tests establish, including the symmetry-based exclusion at 40 examples, we ask that you reconsider the informativeness and overall ratings.

Reviewer_zcWcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when behavioral tests can rule out computational explanations for in-context learning. For Gaussian linear regression, it constructs an optimal Krylov-space oracle to establish a prediction-error floor for entire families of gradient-style algorithms, including adaptive updates and specified preconditioners. It further uses Gaussian MINRES theory and symmetry analysis to explain how context length affects the strength of such tests and to distinguish useful computational differences from symmetry-dependent errors. Experiments on two independently trained 12-layer Transformers show that both outperform the best 12-update algorithm families with shorter contexts n=22, but not with longer contexts n=40, despite improved prediction accuracy. At n=40, approximately 96–97% of recovered coefficient error is symmetry-dependent.

Critical strengths

This paper provides a quantitatively interpretable test that can potentially rule out algorithm families proposed to explain the in-context learning capabilities of Transformers.

Critical weaknesses

My main concern with this paper is whether it offers any new knowledge or conclusions. In fact, I doubt anyone would expect that an actually trained L-layer transformer performs exactly L steps of gradient descent. If an L-layer transformer really did implement L steps of some classical algorithm, there would be no need to train such a model at all; we could simply run the classical algorithm for L steps.

The phenomenon the authors observe is not surprising either: the transformer outperforms the oracle when n is small and underperforms it when n is large. Training a transformer amounts to fitting the distribution of the training dataset, so the model inevitably picks up prior information about that distribution. (For example, both [1] and [2] found that transformers learn the covariance of the training data during training.) This prior can be viewed as a form of inductive bias, and with an appropriate inductive bias, it is unsurprising that a transformer can outperform classical algorithms when only a few samples are available. Conversely, once enough in-context information is available, this additional inductive bias may become ineffective or even harmful.

For these reasons, I do not find the paper's experimental findings surprising, and I do not find the paper provides any new knowledge or conclusions.

Refrences: [1]: Zhang, Ruiqi, Spencer Frei, and Peter L. Bartlett. "Trained transformers learn linear models in-context." Journal of Machine Learning Research, 2024.

[2]:Shen, Wei, Ruida Zhou, Jing Yang, and Cong Shen. "On the training convergence of transformers for in-context classification of gaussian mixtures." ICML, 2025.

Questions for authors

Could the authors provide any new insights or conclusions can be drawn from the methodology proposed in this paper?

1 comment

Thank you for asking what the methodology lets us conclude beyond a comparison with ordinary gradient descent.

First, it tells us how close an entire family can come to explaining the model’s predictions. At 22 examples, Table 11 bounds the public checkpoint’s normalized RMS distance to every budget-12 unpreconditioned predictor below by 0.0884, while bounding its RMS error against the true target above by 0.06890 on the same simultaneous confidence event. So no predictor in that family matches the model as closely as the model matches the target. The result also does not depend on equating one layer with one GD step: both 12-layer models beat the budget-16 unpreconditioned oracle at 22 examples (Table 1).

Second, a longer context can improve prediction while making the risk-based test less informative about computation. From 22 to 40 examples, the public model’s RMS error falls from 0.06321 to 0.02402, but the budget-12 unpreconditioned oracle’s floor falls from 0.17114 to 0.00335 (Tables 3 and 14). The model predicts better, yet prediction error by itself can no longer rule out the family. The symmetry test still excludes the same unpreconditioned class for both models at 40 examples (Proposition 5; Table 12). What disappears at longer context is the evidence from prediction error.

On learned priors, we agree that training can encode information about the data distribution. In the isotropic population case of Zhang et al. (2024, Theorem 4), under their stated assumptions, the learned correction is a positive scalar that leaves the Krylov span unchanged (Appendix B.7, lines 874–879). The oracle already optimizes over all scalar combinations in that span for each prompt, so that specific correction cannot explain performance below the floor. A learned prior that supplies new directions remains a possible explanation for the departure.

In light of these results, we ask that you reconsider the informativeness and overall ratings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.