- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper shows the approximation theory for Transformers. This paper list lots of theorem, but it seems many of the theories are much related to the previous work. It is hard to distinguish which of these theories are the contribution of this paper.
Critical strengths
There are lots of mathematical results in this paper.
Critical weaknesses
Presentation. In the paper writing, this paper just put lots of the results together, but do not claim their relationship, how to extend one result to anther, and the differences with the previous work. Therefore, I can hardly find the real contribution of this paper and what is important this paper want to tell us. Although this paper contains lots of mathematical results, I can hardly find some interesting thoughts for human researchers.
Limited discussions. Besides, Transformers have been extended to many variants for several years---like causal attention, linear attention and sparse attention, which may replace the softmax attention. These variants are not discussed, so the title of this paper seems not suitable.
Questions for authors
- What is the contribution of this paper?
- What is different from the previous work on the approximation theory for Transformers? How important is this paper?
- Can the analyses extend to other attentions? such as causal attention, linear attention and sparse attention.
3 comments
Questions For Authors: 1 What is the contribution of this paper?
Our main contribution is to show that sequence-to-sequence transformers can inherit classical neural network approximation theory (including universality and rates).
To this end, the key new technical tool is the exact realization theorem (Theorem 3.3); this shows that sequence-to-sequence transformers exactly realize prescribed families of shallow networks with shared weights and affine positional biases. The main technical idea of the proof is to use attention to gather information and feed it to the feedforward layers (see the explanation at the beginning of the proof of Th. 3.3).
This exact realization lets us transfer classical neural network approximation theory to transformers, using the following very natural strategy: given a target , we first adapt (via technical arguments in Appendices B.2, B.3, and B.5) classical network approximation results to construct an approximating family of this form. We then realize this family exactly with a transformer. We show that this procedure preserves universality and rates.
As a consequence, we provide the first comprehensive approximation theory for a general class of transformers, with results that parallel those in classical neural network approximation theory: we prove transformer analogues of Hornik's four universal approximation theorems, derive , Sobolev, , and Barron rates, and show that the rate is optimal under continuous parameter selection. These are among the many material results that the Referee noted under Critical Strengths.
As mentioned in the previous comment, to address the Referee's point, we have made the two steps of the transfer argument more explicit in the revised introduction, Section 3, and Appendices B.2--B.3, B.5.
2 What is different from the previous work on the approximation theory for Transformers? How important is this paper?
Previous approximation theory for transformers was incomplete: previous results typically gave guarantees for particular architectures, target classes, or error criteria, while several basic questions remained open, including (but not only) universality and rates for derivative approximation.
Our exact realization theorem lets us transfer classical neural network approximation theory to positional transformers. In particular, we extend compact-uniform universality to general continuous nonpolynomial activations and universality to arbitrary finite Borel input measures and bounded Borel measurable activations (Theorems 4.1--4.2). To the best of our knowledge, Theorems 4.3 and 4.4 provide the first universal approximation theorems for transformers that simultaneously approximate functions and their derivatives in the compact-open topology and in weighted Sobolev norms.
Our results in Section 5 provide the first Sobolev, , and Barron derivative-approximation rates for transformers by transferring classical NN rates. For general smooth nonpolynomial activations, Theorem 5.3 with gives the rate for targets whose reduced components have regularity, where is the largest number of active input tokens at any output position. Theorem 6.1 proves optimality of this exponent for continuous parameter selection.
We have clarified these points by providing a revised comparison in the introduction and by adding a comparison with selected literature in Appendix E.
How important is this paper?
Besides the breadth and usefulness of our results, the significance of our paper, in our view, is that it provides a very natural approach for transformer approximation theory. Proving universality and rates (including derivative approximation) for transformers can be technical and difficult. Indeed, many fundamental universality results and rates had remained unproven, as shown in our literature review. Our paper shifts this approach to transferring the corresponding classical neural-network theory as a whole. One can focus on proving exact realization for suitable families of NNs, which our paper identifies. We have clarified this in the introduction.
- Can the analyses extend to other attentions? such as causal attention, linear attention and sparse attention?
We have extended our results in the revised manuscript to attention with a mask (see Appendix A.1 for masks and Definition 2.1(ii) for the corresponding transformer architecture). This includes, in particular, causal, sparse (as suggested by the Referee), local, and full attention, illustrating the flexibility of the framework. The approximation results apply to targets compatible with the chosen mask (Definition A.3 and Remark A.4 on mask compatibility). Extending the rates to masked attention required a new transfer argument (Steps 1--4 of Appendix B.3), because the original translation argument uses the full input at every output position. We retain the simpler translation argument in Appendix B.5 for the full-attention rate extensions to deep networks and equivariant targets in Appendices C and D, respectively.
Furthermore, in Remark B.9, we identify linear attention as a possible extension of our framework and leave it for future work.
We are grateful to the Referee for carefully reading the manuscript and for the comments. These helped us, in particular, to clarify how the exact realization theorem connects the paper's results and prompted us to extend the theory to masked attention. We have addressed these comments in the revised version, with the main changes highlighted in red. As a result, our paper now contains a clearer presentation and stronger results, and we are grateful to the Referee for this.
Below, we address all the points raised and we specify the corresponding changes in the paper.
Our main contribution is to show that sequence-to-sequence transformers can inherit classical neural network approximation theory (including universality and rates, i.e., all the results that the Referee noted under Critical Strengths).
To this end, the key new technical tool is the exact realization theorem (Theorem 3.3, i.e., transformers exactly realize suitable neural networks), from which the approximation results can be inherited. In particular, given a target f, we first adapt classical neural network approximation results (in Appendices B.2, B.3, and B.5) to construct an approximating family in a form compatible with Theorem 3.3. We then realize this family with a transformer via Theorem 3.3.
In our original submission, we had outlined this relationship between exact realization and the approximation results in the introduction, Section 3, and Appendices B.2--B.3. To address the Referee's point, we have made the two steps of the transfer argument more explicit in the revised introduction, Section 3, and Appendices B.2--B.3, B.5.
We have explained these points in detail in our answers to Questions 1 and 2 below, including a comparison with the literature.
We extended the results to softmax attention with a fixed mask, including causal and fixed sparse attention, as suggested by the Referee. Furthermore, in Remark B.9, we identify linear attention as a possible extension of our framework and leave it for future work.
Our answer to Question~3 explains these points in detail.