acceptodds
Under review as a conference paper at ICLR 2027

A DEPTH–WIDTH SEPARATION FOR QUADRATIC IN-CONTEXT LEARNING WITHPROTECTED-LABEL BILINEAR TRANSFORMERS

Abstract

Quadratic in-context regression has task coefficients for inputs in , although each prediction is a scalar. Can depth reduce the token width needed to predict these tasks? We study quadratic regression with Gaussian inputs using a bilinear Transformer whose attention can access labels but whose feed-forward maps cannot read or modify them. For one or two blocks at width , the minimum achievable relative risk converges to one, uniformly over context length. Three blocks suffice at width : an explicit network reaches relative risk at most with context examples. Thus the minimum width for fixed relative risk below one falls from to between two and three blocks. The construction evaluates the quadratic projection-kernel estimator without storing all quadratic features. Attention applies a context-dependent matrix to the query, and a later block completes the prediction using -dimensional states. This shows that sequential computation through context can replace explicit storage of a quadratic feature representation.

Primary area:
learning theory
Keywords:
in-context learning, transformer expressivity, depth-width separation, quadratic regression, kernel methods, learning theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7XuoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a deeper model requires smaller width to learn quadratic in-context learning regression tasks. The authors consider Gaussian inputs, Gaussian tasks, noiseless labels, and a transformer with blocks of bilinear feedforward and linear attention sublayers. The feedforward sublayers cannot read the labels directly. They show that a model with one or two blocks requires width proportional to d^2 to achieve a fixed improvement over predicting zero. They then construct a three-block model with width 2d+5 that achieves small relative prediction error when the context is sufficiently long. By construction, the attention applies a matrix defined by the context to the query and then a final feedforward sublayer uses that to predict the label. This way, the models do not need to store all the quadratic features of each token. Finally, the risk of the 3-block architecture is compared to the Bayes risk.

Critical strengths
  1. The paper provides a clear improvement caused by model depth. It shows that a model with three blocks requires less width to in-context learn the quadratic regression tasks than a model with one or two blocks.

  2. The authors show that the limitation of 1 or 2-block model still holds with arbitrary context length.

  3. They show a mechanism by construction on how the model avoids to store quadratic features and only require a linear width for 3-block model.

Critical weaknesses
  1. The contribution of the paper is a narrow extension of [1]. The data model, task distribution, model architecture, and kernel estimation approach are the same as [1]. The consequences of the central contribution of the paper are presented repeatedly. For example, discussion on equation (5), item 1 and 2 of theorem 3.1, and corollary 6.1 all discuss the same thing. A shorter presentation might help more with the readability. [1] Sun, H., Jadbabaie, A., and Azizan, N. On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning. ALT 2026.

  2. The paper includes algebraic calculations, but no synthetic or real-world experiments are included. Experiments with synthetic data, and simple models with varying width and depth would help to assess whether a trained model shows the width requirement separation discussed in the paper.

  3. As mentioned in the paper, the results rely on protected-label assumption. If the feedforward layers have access to the labels, a model with two blocks and linear in d width would suffice as discussed in Appendix B.1. However, the practical motivation of considering this assumption is not clear.

  4. Same Z is used as the input for the attention block and the bilinear feedforward sublayer in equation (2)-(3), which might be confusing as the output of feedforward sublayer is the input of the linear attention.

Questions for authors
  1. Is there a practical motivation for considering the protected-label assumption or is the reason purely theoretical?

  2. When the context length equals the number of task coefficients, the Bayes predictor can successfully recover the task. However, the 3-block model requires d^2/\epsilon order of context length to do so. Can this gap between the two models decrease by a different construction of the 3-block model?

  3. Can conventional training methods discover the constructed model? It would be helpful to include an optimization analysis on how such a model can be trained.

Reviewer_P75eOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the depth–width tradeoff in Transformers for in-context learning of quadratic regression tasks. Specifically, it considers noiseless quadratic functions with Gaussian inputs and coefficients, where the number of task coefficients is Θ(d2)\Theta(d^2). The analysis focuses on a bilinear Transformer architecture, in which each block consists of a bilinear feed-forward layer followed by linear attention. Importantly, the architecture imposes a restrictive protected-label assumption, under which feed-forward layers cannot directly access or modify labels, while attention layers can.

The main theoretical contribution is a sharp separation between two and three Transformer blocks. The authors prove that one or two block Transformers require width Θ(d2)\Theta(d^2) to achieve any fixed relative risk below one, regardless of context length, where relative risk one is achieved by the trivial predictor that always outputs zero. In contrast, they construct an explicit three-block Transformer of width 2d+5 that achieves relative risk at most ϵ\epsilon using O(d2/ϵ)O(d^2/\epsilon) context examples.

The construction implements a quadratic projection-kernel estimator by using attention to compute a context-dependent matrix–vector product without explicitly storing the quadratic-dimensional matrix. The paper also analyzes the estimator's finite-context risk and compares its performance with the Bayes-optimal predictor.

Critical strengths

Strong and theoretically interesting separation. The paper establishes a sharp depth-width separation, showing that adding just one Transformer block can reduce the required width from quadratic to linear in the input dimension. I found the key insight particularly interesting, as it is both simple and useful: although the quadratic task has Θ(d2)\Theta(d^2) coefficients, the Transformer can avoid explicitly storing them by using attention to compute a context-dependent matrix–vector product and then combining the result with the query in a subsequent block. The lower bound is also nontrivial.

Critical weaknesses
  1. The separation relies heavily on a restrictive architectural assumption: The result critically depends on the protected-label restriction, which prevents feed-forward layers from directly accessing labels. As shown in the paper, removing this restriction allows a two-block Transformer of linear width Θ(d)\Theta(d) to implement the same estimator. Since standard Transformers do not impose this restriction, it is unclear whether the separation reflects a fundamental property of Transformer depth or this specific architectural choice. Moreover, the use of bilinear feed-forward layers before attention differs from standard Transformers, further limiting the generality of the result.

  2. Lack of experimental validation: The paper is entirely theoretical, and it would be interesting to investigate whether gradient descent can learn the proposed solution in a three-block Transformer, under a suitable training distribution. Additionally, it would be valuable to examine empirically whether similar depth-width phenomena arise in more realistic Transformer architectures, with some modifications to the task setting.

Questions for authors
  1. Do you expect a meaningful depth-width separation to hold without the protected-label assumption, with a slight modification to the setting?
Reviewer_BBpYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers a question of in-context quadratic regression, i.e. trying to implicitly learn linear combinations of 11, xix_i and xixjx_i x_j i.e. up to 2nd order features of the dd-dimensional data from examples. The paper then focuses on the question of if internal representation dimension (i.e. width) really needs to also be quadratic in order to sufficiently perform at this task in-context.

Critical strengths

This is a nice task to study in the ICL setting particularly as it builds on other works primarily focusing on linear regression, where width bottleneck is not so much an issue. Here the authors clearly and succinctly, in my opinion, lay out the width-scaling problem arising from this quadratic ICL task as well as their results. I cannot find anything obviously wrong, given my current understanding (and without going through the appendix) of the constructions given.

Critical weaknesses

My main criticism of this paper is, while one of its strengths is precisely this constructive approach, I'm not convinced of its generality. The main paper figures are a conceptual illustration (Figure 1, which I criticise below) and an analyitical plot of some theory regions (figure 2); neither of these are experimental or simulations at all. I understand the main contribution of this paper is it's theoretical construction but I would argue this still warrents at least some empirical verification in the main text.

Complementary to the above issue, I'm concerned how transferable the results are outside of the very specific architecture discussed.

Finally, Figure 1 is badly designed and does not add much clarity in my opinion, particularly as the internal text of the figure is extremely small and sometimes text is even overlapping in places.

Questions for authors

No further questions beyond answering those stated in the above limitations

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.