Universal Consistency of Transformers as Functional Regressors

Transformers are an indispensable component of modern foundation models, yet their theoretical understanding for functional regression on non-Euclidean domains remains underexplored. While the universal approximation properties of Euclidean Transformers have been extensively studied approx_tran, their statistical consistency as functional regressors remains less understood, particularly beyond Euclidean settings. Motivated by recent advances in hyperbolic neural networks and the intrinsic hierarchies of data such as text, images, and graphs, we develop a statistical framework for analyzing Transformers in hyperbolic spaces. We establish universal approximation results for broad classes of sequence-to-sequence functional maps and prove universal statistical consistency for least-squares functional regression without imposing a parametric model specification. We further derive stochastic error bounds for the corresponding empirical estimators, yielding an asymptotic rate of $\mathcal{O}(t^{-1/2d})$, where $t$ denotes the number of input tokens and $d$ the input embedding dimension. Our analysis explicitly characterizes the curse of dimensionality induced by the ambient embedding dimension, providing a quantitative account of the statistical complexity of nonlinear attention-based Transformer architectures. The Euclidean setting is recovered as a special case of the proposed framework. Experiments on real datasets with both continuous and categorical responses empirically support our theoretical results and demonstrate the practical relevance of hyperbolic representations for functional sequence-to-sequence regression.