- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper develops a comprehensive and timely quantitative analysis of transformer neural operators between separable Hilbert spaces, which is especially relevant for partial differential equation (PDE) foundation models. The target operators belong to a Frechet-Holder class of operators. The analysis makes three contributions: approximation analysis, generalization analysis of empirical risk minimization (ERM) under squared loss, and minimax lower bounds that show ERM is optimal. The authors allow for subexponential noise and data, which is more general than previous work. Short but well-executed numerical experiments further verify the theory and boost the impact of the paper.
Critical strengths
While the paper is highly technical, it is well written and easy to follow. The analysis is highly complete. It includes positional encodings, MLPs, multi-head attention, and general bounds uniform over the target, noise, and input distributions. The fact that the depth is only 2 is interesting, contrasting with other work that requires depth scaling with the approximation error or sample size. Another impactful contribution is the discretization error in Sec 4.2 that is propagated through the generalization analysis, showing that the discretization degrees of freedom can be chosen exponentially smaller than the sample size n of input-output functions.
Critical weaknesses
A minor criticism is the reduction from Hilbert spaces X and Y to . This is of course theoretically justified, but practically it may be hard to identify and efficiently compute orthonormal bases of the Hilbert spaces. It also fixes the countable sequence space interpretation of transformers instead of the continuum uncountable integral representations in other work on transformer neural operators.
Questions for authors
-
Would use of layer norm complicate the analysis?
-
It could be helpful to remark that clipping can be realized as an MLP?
-
I may have missed this, but do any of the assumptions require that the input distribution is genuinely supported on an infinite-dimensional space? Related work by P. Grohs, S. Lanthaler, and M. Trautner. "Theory-to-practice gap for neural networks and neural operators" require an infinite-dimensional hypercube condition.