- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the representational power of Transformer models in nonlinear in-context learning tasks. Specifically, the paper shows that Transformers can solve Gaussian-kernel ridge regression with an explicit construction that approximates a pre-conditioned Richardson iteration. This construction shows that the accuracy of the resulting Transformer model scales with the MLP hidden width on the order of and the number of layers on the order of . The paper concludes with empirical study that directly trains a GPT-2 style model on the Gaussian-kernel ridge regression task and demonstrates that the theoretical finding can translate to a more practical setting.
Critical strengths
I find this paper well-written and enjoyable to read. The notations are clean and the main theorem statement is easy to understand.
The technical contributions of this paper is somewhat reasonable for a premier conference paper (albeit at the lower end of the spectrum) , as it contributes to the understanding of the standard softmax attention without attaching restrictive assumptions. The results of this work clearly demonstrated the interplay between the softmax activation and MLP layers in enabling a nonlinear in-context learning task.
Critical weaknesses
While I don’t think there are any fatal issues with this paper, there are a places where there is some room for improvement.
I feel that the proof outline given in Appendix C.1 should be moved to the main text. That part of proof has a lot of details that would be helpful for the readers’ understanding.
-
The definitions of softmax on lines 77 and 195 do not match. I think they are equivalent if the norm features are included. So, the discussion about this augmentation (on line 254) should be brought forward and displayed more prominently.
-
The attention masks are not specified in the main text. There should be a short discussion in the main text emphasizing that they are non-causal and only used to mask out the dummy and test tokens.
-
The construction is far from economical and quite wasteful, where about half of the attention layers would have to be zeroed out if the construction is put into the standard Transformer blocks consisting of alternating attention and MLP blocks. I would be interested to see if the authors could provide some concrete evidence or justification for such choices.
-
The proof is heavily tied to the correspondence between the Gaussian kernel and the softmax function. I am interested in seeing if the work could be extended to more general classes of kernels.
-
While the results are fairly interesting, the technical principles in this paper seem to be heavily borrowed from Bai et al (Neurips 2023). In light of this, I think it is most appropriate to recommend a weak accept.
Questions for authors
I posed my questions as points in the above section.
4 comments
W5: Relation to Bai et al. (2023)
While the results are fairly interesting, the technical principles in this paper seem to be heavily borrowed from Bai et al (Neurips 2023). In light of this, I think it is most appropriate to recommend a weak accept.
We agree that both papers use explicit transformer constructions, but there are several important design differences. Bai et al.’s linear-regression constructions use ReLU attention, whereas we use standard softmax attention for Gaussian KRR.
Their general gradient construction implements ReLU approximation terms through attention heads. Our construction uses one softmax attention head and controls scalar approximation accuracy through MLP width. This choice is motivated by practical architectures with relatively few heads but wide MLPs (for example, 8 heads and MLP width 1024 in our experiments).
In our construction, using MLPs for tokenwise approximation lets us reduce the required arithmetic to approximations of smooth one-dimensional functions (the square, reciprocal, and ). Explicit linear splines then achieve error with MLP width on fixed intervals. Bai et al.’s general bound for multivariate smooth functions yields a sufficient ReLU-attention head count of in their general gradient construction.
Bai et al.’s linear-regression construction uses an attention-only transformer to implement exact gradient steps. Our Gaussian KRR construction uses both attention and MLP layers, making their complementary roles explicit in implementing inexact preconditioned Richardson updates.
In addition to these theoretical distinctions, we would like to emphasize that the experimental study is a second main contribution. Our theory establishes that transformers can approximate Gaussian KRR. The experiments address the complementary question of whether and how KRR behavior emerges in GPT-2-style models trained without a prescribed solver. Studying both predictions and induced weights across depth helps reveal how this agreement develops, while the controls help assess whether it reflects the learned model’s behavior.
We hope the reviewer will consider both the theoretical and experimental contributions in the overall assessment of the paper.
Reference
Bai et al. (2023). Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. NeurIPS.
We sincerely appreciate your constructive comments, which have helped us improve our work. If our responses have addressed your concerns, we would appreciate it if you could revisit your assessment in light of these updates, and we welcome any further questions or feedback.
Thank you for your explanations. I will maintain my score.
Thank you for your comments. I will maintain my score.
We thank the reviewer for the careful reading and the positive assessment. We have revised the draft to address the concerns below. The revisions in the draft are marked in red.
W0: Proof outline
We added further details from Appendix C.1 to the construction overview after the theorem in Section 4 (lines 292--303). We hope these additions help readers understand the construction while keeping the main text concise. We welcome the reviewer’s feedback on whether the added details are sufficient.
W1: Softmax attention and squared-norm features
Definition 1 (lines 189--196) gives the formal definition of softmax attention. The expression in the introduction is a special case in which softmax normalization yields the row-normalized Gaussian kernel matrix. We now state explicitly that this specialization uses squared-norm features and suitable query and key matrices (Introduction, lines 73--76).
W2: Attention masks
We added the mask description to Section 4 (lines 293--294). The construction uses noncausal masks that exclude only dummy or test tokens, or both.
W3: Economy of the construction
We thank the reviewer for this question. Our MLP uses two linear maps with one ReLU activation layer, which follows the standard transformer feedforward structure.
In our staged construction, some computations take the output of an earlier MLP as their input. Since we retain one hidden layer per MLP, we implement these dependent operations in separate transformer blocks. The additional stages need no further interaction between tokens, so we use MLP-only blocks.
We believe that deeper MLPs could combine some stages into fewer blocks. In this work, we retain one hidden ReLU layer per MLP to follow the standard transformer feedforward structure. Our main goal is to establish an approximation guarantee within this architecture and explain the separate roles of attention and MLP layers. We agree that more economical constructions within the same architecture may be possible.
W4: Other kernels
We thank the reviewer for this suggestion and agree that extending the result to other kernels would be valuable. We chose the Gaussian kernel because its close connection to softmax allows us to give an explicit construction. We believe that a similar approach may work for exponential dot-product kernels. For more general continuous kernels, it may be worth exploring whether approximating these kernels with Gaussian basis functions by multiple attention heads could help extend our softmax construction. The feasibility of this approach and its effect on the final KRR prediction would need further analysis.