- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how a deeper model requires smaller width to learn quadratic in-context learning regression tasks. The authors consider Gaussian inputs, Gaussian tasks, noiseless labels, and a transformer with blocks of bilinear feedforward and linear attention sublayers. The feedforward sublayers cannot read the labels directly. They show that a model with one or two blocks requires width proportional to d^2 to achieve a fixed improvement over predicting zero. They then construct a three-block model with width 2d+5 that achieves small relative prediction error when the context is sufficiently long. By construction, the attention applies a matrix defined by the context to the query and then a final feedforward sublayer uses that to predict the label. This way, the models do not need to store all the quadratic features of each token. Finally, the risk of the 3-block architecture is compared to the Bayes risk.
Critical strengths
-
The paper provides a clear improvement caused by model depth. It shows that a model with three blocks requires less width to in-context learn the quadratic regression tasks than a model with one or two blocks.
-
The authors show that the limitation of 1 or 2-block model still holds with arbitrary context length.
-
They show a mechanism by construction on how the model avoids to store quadratic features and only require a linear width for 3-block model.
Critical weaknesses
-
The contribution of the paper is a narrow extension of [1]. The data model, task distribution, model architecture, and kernel estimation approach are the same as [1]. The consequences of the central contribution of the paper are presented repeatedly. For example, discussion on equation (5), item 1 and 2 of theorem 3.1, and corollary 6.1 all discuss the same thing. A shorter presentation might help more with the readability. [1] Sun, H., Jadbabaie, A., and Azizan, N. On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning. ALT 2026.
-
The paper includes algebraic calculations, but no synthetic or real-world experiments are included. Experiments with synthetic data, and simple models with varying width and depth would help to assess whether a trained model shows the width requirement separation discussed in the paper.
-
As mentioned in the paper, the results rely on protected-label assumption. If the feedforward layers have access to the labels, a model with two blocks and linear in d width would suffice as discussed in Appendix B.1. However, the practical motivation of considering this assumption is not clear.
-
Same Z is used as the input for the attention block and the bilinear feedforward sublayer in equation (2)-(3), which might be confusing as the output of feedforward sublayer is the input of the linear attention.
Questions for authors
-
Is there a practical motivation for considering the protected-label assumption or is the reason purely theoretical?
-
When the context length equals the number of task coefficients, the Bayes predictor can successfully recover the task. However, the 3-block model requires d^2/\epsilon order of context length to do so. Can this gap between the two models decrease by a different construction of the 3-block model?
-
Can conventional training methods discover the constructed model? It would be helpful to include an optimization analysis on how such a model can be trained.