acceptodds
Under review as a conference paper at ICLR 2027

When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers

Abstract

Transformers perform computations through many successive attention and feedforward blocks, allowing them to learn complex correlations between a large collection of coupled variables. When does stacking successive attention-feedforward computations over many layers provide a computational advantage over a single transformer block? We address this question by examining in-context learning in generalized linear attention transformers. We first introduce a general theory of distributed inference in such transformers, subject to constraints on communication and depth. We show that such systems can exploit internal representations (`function vectors') to infer a latent context variable at increasingly finer scales over its layers. For an in-context linear regression task, the theory predicts that while one-layer transformers without feedforward blocks are optimal for Gaussian priors over the context variable, multi-layer transformers are superior for non-Gaussian, tree-like priors. Trained linear attention transformers reproduce quantitative predictions from the theory. Using causal key-patching experiments, we verify that function vectors in intermediate layers mediate adaptive routing of information. Our results suggest that depth and feedforward blocks enable transformers to implement adaptive inference, and this is advantageous when the distribution over latent variables has hierarchical structure.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
In-context learning, transformers, function vectors, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rT4GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study the role of depth in in-context learning via the adaptive inference. For the in-context linear regression, they analyze the adaptive and non-adaptive inference via generalized linear attention transformers under the Gaussian and tree prior. Their analysis predicts that depth provides no advantages under the Gaussian prior, while it can be beneficial under the tree prior. They also empirically support for these predictions.

Critical strengths

The strength of this paper is to interpret the role of MLPs and attention in adaptive inference. MLPs determine which information should be extracted next, while attention extract the information from the context. This interpretation is interesting. The experiments provide good empirical support for the theoretical results.

Critical weaknesses

The theory focuses on in-context linear regression with linear attention. Although the tree prior is considered as a non-Gaussian example, it is still a specific case. Thus, it is not clear if the role of MLPs and attention discussed in this paper can also be obtained in more practical setting, such as nonlinear in-context learning, standard softmax attention, and other non-Gaussian prior.

Questions for authors

Is there any intrinsic difficulty in extending the analysis beyond linear regression, linear attention, and other non-Gaussian prior ? In particular, could the same interpretation of depth and the role of MLPs and attention be extended to these case ?

2 comments

Thank you for your kind comments on our work, we hope that the following has adequately addressed your questions:

  1. We believe that extending this study beyond linear regression should not be a theoretical problem, and it is merely a computational choice for this study. Note that the theory presented in Section 2 applies to any distribution that satisfies Equation 1. It is also known, by de Finetti's theorem, that exchangeable random variables are conditionally independent given a latent variable, i.e., their joint distribution can be written in the form of Equation 1. Thus, the theory applies to any setting where the observations are exchangeable, which includes linear regression.
  2. One specific example of an extension is to define a suitable kernel ϕ\phi so the attention operations act on (ϕ(x),ϕ(y))(\phi(x),\phi(y)) instead of (x,y)(x,y). This kernel may be learned or it may be judiciously chosen as some suitable polynomial or Fourier basis set (for example). We chose to stick with linear regression as the number of basis elements (and thus the model dimension) required grows quickly with the degree of the polynomials in consideration.
  3. Extending this beyond linear attention is not simple in the theoretical sense, and this is a current area of study for us. However, we were able to confirm with new experiments in Appendix C that the observed phenomenon of advantages from depth in the tree prior persists for softmax attention.
  4. We have considered two specific priors in our study, but we believe the message extends beyond these. Specifically, we find that Gaussian priors are special in that they are the only ones which do NOT confer any advantage through depth. Any non-Gaussianity present in the prior is exploitable, and we have demonstrated this using a tree prior for maximal impact, but we expect this phenomenon to similarly show up in other non-Gaussian priors as well.

Thank you very much for the clarification. I would like to keep my current score as weak acceptance.

Reviewer_Ad3XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents the concept that transformers with linear attention can effectively use depth to capture hierarchical, "tree-like" information in in-context learning. They support this with a theoretical framework showing how information gathered in earlier layers can guide the information that subsequent layers extract. With this framework, they predict that, for an in-context linear regression task, while one-layer transformers without feedforward blocks are optimal for Gaussian priors over the context variable, multi-layer transformers are superior for non-Gaussian, tree-like priors. Their experiments support this prediction.

They further conduct causal key-patching experiments to better illustrate how this routing mechanism works through intermediate layers. This analysis is currently limited to a particular class of linear self-attention transformers, but the authors extend this intuition to transformers in general, suggesting that depth and feedforward blocks enable transformers to implement adaptive inference. This is advantageous when the distribution over latent variables has hierarchical structure.

Critical strengths

(1) The idea that depth helps transformers capture hierarchical information in ICL is intuitive and understandable, but not obvious. That observation alone is really interesting. In particular the demonstration that there are times when depth doesn't always provide an advantage further highlights this point. It also presents an interesting look at how we can start designing future domain specific models so that for particularly hierarchical datasets you'd want deep models with small communication channels.

(2) The theory is solid and clearly written providing good insight into the problem. The experiments are also appropriately chosen to highlight the key points behind the theory.

Critical weaknesses

(1) The author acknowledges that the theory doesn't carry over exactly to softmax attention and one wouldn't expect the experiments to carry over exactly. However, they also don't seem to provide any experiments supporting the intuition in regular softmax transformers. This would greatly strengthen the paper if added.

(2) The experimental architecture is specifically designed for adaptive routing. This is reasonable for testing the theory, but it's not clear whether the same mechanism would naturally emerge in more standard transformer architectures/ regular LSA. Do any of these observations persist?

Questions for authors

(1) What happens when running similar experiments with regular softmax attention? Are there any other experiments that are similar and can support this intuition with regular softmax attention?

(2) Does the same adaptive routing mechanism emerge in standard linear self-attention transformers? How much do the results depend on this design?

2 comments

Thank you for your kind comments on our work, we hope that the following has adequately addressed your questions:

  1. As the current theory applies to linear attention, it is unclear whether theoretical predictions will still hold true for softmax attention. To test this, we performed numerical experiments with softmax attention (Appendix C, specifically Fig. 5(c)). The results provide experimental evidence that depth does indeed present an advantage for the tree prior. We have also done the same for conventional linear attention in this Appendix, confirming that our communication-constrained architecture is not specifically required for this phenomenon.
  2. We also implemented the causal patching experiments (Fig. 3(d)) for the case of conventional linear attention. The results, presented in Appendix C (Fig. 5(d)), confirm that the adaptive routing persists even in this architecture.

Please also refer to the general comment we have posted above for a full summary of changes made.

Thank you for your rebuttal, this has appropriately addressed my concerns, I will maintain my current score.

Reviewer_6aidOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

A major question in transformer models is the role of depth. In this study, the authors explore depth in linear attention models during an in context learning task. The paper performs a theoretical analysis of function vectors during depth-wise calculations. This analysis indicates that for Gaussian priors a one-layer model is optimal, whereas for tree-like priors / data multi-layer transformers are better. These expectations are then recapitulated in experiments training linear attention transformers.

Critical strengths

I believe that this paper has significant strengths. The writing was excellent, the motivation clear, the analysis sound, and the experiments compelling. I enjoyed reading the paper and felt that I learned about the topic.

Critical weaknesses

The primary question that I have regards the novelty relative to prior results, but this is not my area of research so I would defer to other reviewers on this topic. I believe that it may be valuable for the authors clarify the similarities and differences to prior work outside of the recent studies with Transformers:

Braun, Pokutta, and Xie, “Info-Greedy Sequential Adaptive Compressed Sensing” (2015), where the authors study adaptive sensing in a non-Transformer context

Soni and Haupt, “On the Fundamental Limits of Recovering Tree Sparse Vectors from Noisy Linear Measurements”

Questions for authors

Is anything known about how softmax attention will modify these results? I understand that the authors may not be able to say very much theoretically, but even some experimental indication would be helpful for the reader to gauge the relevance of this phenomenon to the architectures that we actually care about. We have found that results can be qualitatively different in the scaling behavior between linear attention and softmax attention.

2 comments

Thank you for your kind comments on our work, we hope that the following has adequately addressed your questions:

  1. As the current theory applies to linear attention, it is unclear whether theoretical predictions will still hold true for softmax attention. To test this, we performed numerical experiments with softmax attention (Appendix C, specifically Fig. 5(c)). The results provide experimental evidence that depth does indeed present an advantage for the tree prior.
  2. We have also inserted a brief discussion in Sec 1.1 - Related Work pertaining to how our work fits in the field of adaptive sensing and sequential experimental design, which we hope better contextualizes the nature of the questions addressed in this paper.

Please also refer to the general comment we have posted above for a full summary of changes made.

I appreciate that the authors have done some new experiments on softmax attention (and explored weakening assumptions within the linear attention framework). If I were able to I would now give this manuscript a 3.5.

Comments on the paper

1 comment

We thank the reviewers for their kind comments and many helpful suggestions. Here is a summary of changes that we have made in response to the suggestions:

  1. New experiments on conventional linear self-attention (LSA) and softmax self-attention (SSA) transformers in Appendix C. These confirm that our findings regarding the advantages of depth and its dependence on prior structure persist (qualitatively) even in regular architectures and are not limited to the constrained architecture that we have used to draw emphasis to the routing performed by the MLPs.
  2. Additional paragraph in Sec 1.1 - Related Work, pertaining to adaptive inference and prior work in adaptive sensing. We contextualize our work in this space as a study of how transformers implement such sequential decision-making processes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.